WCF, BasicHttpBinding: Stop new connections but allow existing connections to continue

Question

.NET 3.5, VS2008, WCF service using BasicHttpBinding

I have a WCF service hosted in a Windows service. When the Windows service shuts down, due to upgrades, scheduled maintenance, etc, I need to gracefully shut down my WCF service. The WCF service has methods that can take up to several seconds to complete, and typical volume is 2-5 method calls per second. I need to shut down the WCF service in a way that allows any previously call methods to complete, while denying any new calls. In this manner, I can reach a quiet state in ~ 5-10 seconds and then complete the shutdown cycle of my Windows service.

Calling ServiceHost.Close seems like the right approach, but it closes client connections right away, without waiting for any methods in progress to complete. My WCF service completes its method, but there is no one to send the response to, because the client has already been disconnected. This is the solution suggested by this question.

Here is the sequence of events:

1. Client calls method on service, using the VS generated proxy class
2. Service begins execution of service method
3. Service receives a request to shut down
4. Service calls ServiceHost.Close (or BeginClose)
5. Client is disconnected, and receives a System.ServiceModel.CommunicationException
6. Service completes service method.
7. Eventually service detects it has no more work to do (through application logic) and terminates.

What I need is for the client connections to be kept open so the clients know that their service methods completed sucessfully. Right now they just get a closed connection and don't know if the service method completed successfully or not. Prior to using WCF, I was using sockets and was able to do this by controlling the Socket directly. (ie stop the Accept loop while still doing Receive and Send)

It is important that the host HTTP port is closed so that the upstream firewall can direct traffic to another host system, but existing connections are left open to allow the existing method calls to complete.

Is there a way to accomplish this in WCF?

Things I have tried:

1. ServiceHost.Close() - closes clients right away
2. ServiceHost.ChannelDispatchers - call Listener.Close() on each - doesn't seem to do anything
3. ServiceHost.ChannelDispatchers - call CloseInput() on each - closes clients right away
4. Override ServiceHost.OnClosing() - lets me delay the Close until I decide it is ok to close, but new connections are allowed during this time
5. Remove the endpoint using the technique described here. This wipes out everything.
6. Running a network sniffer to observe ServiceHost.Close(). The host just closes the connection, no response is sent.

Thanks

Edit: Unfortunately I cannot implement an application-level advisory response that the system is shutting down, because the clients in the field are already deployed. (I only control the service, not the clients)

Edit: I used the Redgate Reflector to look at Microsoft's implementation of ServiceHost.Close. Unfortunately, it calls some internal helper classes that my code can't access.

Edit: I haven't found the complete solution I was looking for, but Benjamin's suggestion to use the IMessageDispatchInspector to reject requests prior to entering the service method came closest.

Devereux

Answer 1

Guessing:

Have you tried to grab the binding at runtime (from the endpoints), cast it to BasicHttpBinding and (re)define the properties there?

Best guesses from me:

• OpenTimeout
• MaxReceivedMessageSize
• ReaderQuotas

Those can be set at runtime according to the documentation and seem to allow the desired behaviour (blocking new clients). This wouldn't help with the "upstream firewall/load balancer needs to reroute" part though.

Last guess: Can you (the documention says yes, but I'm not sure what the consequences are) redefine the address of the endpoint(s) to a localhost address on demand? This might work as a "Port close" for the firewall host as well, if it doesn't kill of all clients anyway..

Edit: While playing with the suggestions above and a limited test I started playing with a message inspector/behavior combination that looks promising for now:

public class WCFFilter : IServiceBehavior, IDispatchMessageInspector {
    private readonly object blockLock = new object();
    private bool blockCalls = false;

    public bool BlockRequests {
        get {
            lock (blockLock) {
                return blockCalls;
            }
        }
        set {
            lock (blockLock) {
                blockCalls = !blockCalls;
            }   
        }

    }

    public void Validate(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase) {          
    }

    public void AddBindingParameters(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase, Collection<ServiceEndpoint> endpoints, BindingParameterCollection bindingParameters) {         
    }

    public void ApplyDispatchBehavior(ServiceDescription serviceDescription, ServiceHostBase serviceHostBase) {
        foreach (ChannelDispatcher channelDispatcher in serviceHostBase.ChannelDispatchers) {
            foreach (EndpointDispatcher endpointDispatcher in channelDispatcher.Endpoints) {
                endpointDispatcher.DispatchRuntime.MessageInspectors.Add(this);
            }
        } 
    }

    public object AfterReceiveRequest(ref Message request, IClientChannel channel, InstanceContext instanceContext) {
        lock (blockLock) {
            if (blockCalls)
                request.Close();
        }
        return null;
    }

    public void BeforeSendReply(ref Message reply, object correlationState) {           
    }
}

Forget about the crappy lock usage etc., but using this with a very simple WCF test (returning a random number with a Thread.Sleep inside) like this:

var sh = new ServiceHost(new WCFTestService(), baseAdresses);

var filter = new WCFFilter();
sh.Description.Behaviors.Add(filter);

and later flipping the BlockRequests property I get the following behavior (again: This is of course a very, very simplified example, but I hope it might work for you anyway):

// I spawn 3 threads Requesting a number..
Requesting a number..
Requesting a number..
// Server side log for one incoming request
Incoming request for a number.
// Main loop flips the "block everything" bool
Blocking access from here on.
// 3 more clients after that, for good measure
Requesting a number..
Requesting a number..
Requesting a number..
// First request (with server side log, see above) completes sucessfully
Received 1569129641
// All other messages never made it to the server yet and die with a fault
Error in client request spawned after the block.
Error in client request spawned after the block.
Error in client request spawned after the block.
Error in client request before the block.
Error in client request before the block.

Refrigerator

Your idea on redefining endpoint addresses in interesting. But, it seems there is only a ServiceHost.AddServiceEndpoint. The ChannelDispatcher.Endpoints structure is get-only. Do you know how you would changed the service endpoints after ServiceHost.Open() has been called? – Devereux

Hi. Sorry the things above were just "thinking aloud" suggestions. I'm trying to create a test setup now and have limited (in other words: Not sure if you can use it as-is) success with a Behaviour now. Will update my post in a sec. – Refrigerator

Good idea, I already had a IDispatchMessageInspector behavior to log traffic, so I added a check to see if the host is shutting down, and if so, call Message.Close(). This causes a FaultException on the client, but at least the client is getting the responses for the service calls that came in before the shutdown. The inbound port is still open, unfortunately, but there may be no getting around that. – Devereux

I'm pretty sure there are ways to work around that as well. I'm not a WCF guru myself, but your question made me curious. One way, although far from perfect and probably not what you want, would be to implement the listening yourself, "socket" like. See blogs.msdn.com/drnick/archive/2006/04/18/577936.aspx for an sample. – Refrigerator

Answer 2

Is there an api for the upstream firewall? The way we do this in our application is to stop new requests coming in at the load balancer level, and then when all of the requests have finished processing we can restart the servers and services.

Patsy

No, but we are looking at firewalls that can provide this functionality. Good suggestion, though. – Devereux

+1: this is how I know it as well. Take a service out of rotation on the load balancer and than do whatever maintenance needs to be done. – Expose

Answer 3

My suggestion is to set an EventHandler when your service goes into a "stopping state", use the OnStop method. Set the EventHandler indicating that your service is going into a stopping state.

Your normal service loop should check if this event is set, if it is, return a "Service is stopping message" to the calling client, and do not allow it to enter your normal routine.

While you still have active processes running, let it finish, before the OnStop method moves on to killing the WCF host (ServiceHost.Close).

Another way is to keep track of the active calls by implementing your own reference counter. you will then know when you can stop the Service Host, once the reference counter hits zero, and by implementing the above check for when the stop event has been initiated.

Hope this helps.

Infinitesimal

Thanks, I have already implemented the logic to determine if it is ok to shutdown. The problem is that it is never ok to shut down, because there are always new requests coming in. – Devereux

Answer 4

I haven't implemented this myself, so YMMV, but I believe what you're looking to do is pause the service prior to fully stopping it. Pausing can be used to refuse new connections while completing existing requests.

In .NET it appears the approach to pausing the service is to use the ServiceController.

Bountiful

Yes, what I am trying to do is like a WCF pause. ServiceController is used to interact with Windows services. It does not relate to WCF. – Devereux

Answer 5

Does this WCF Service authenticate the user in any way? Do you have any "Handshake" method?

Aluminate

Authentication is done at the application level. (It's part of the service contract) Credentials are resubmitted with every request. – Devereux

Answer 6

I think you might need to write your own implementation with a helper class that keeps track of all running requests, then when a shutdown is requested, you can find out if anything is still running, delay shutdown based on that... (using a timer maybe?)

Not sure about blocking further incoming requests... you should have a global variable that tells your application whether a shutdown was requested and so you could deny further requests ...

Hope this may help you.

Novelistic

Yes, a helper class is what I implemented. The trick is the blocking of new requests. – Devereux

Answer 7

Maybe you should set the

ServiceBehaviorAttribute and the OperationBehavior attribute. Check this on MSDN

Rior

Answer 8

In addition to the answer from Matthew Steeples.

Most serious load balancers like a F5 etc. have a mechanism to identify if a node is alive. In your case it seems to check whether a certain port is open. But alternative ways can be configured easily.

So you could expose e.g. two services: the real service that serves requests, and a monitoring "heart beat"-like service. When transitioning into maintenance mode, you could first take the monitoring service offline which will take the load away from the node and only shutdown the real service after all requests finished processing. Sounds a bit weird but might help in your scenario...

Expose