Avoiding a ProtocolViolationException following a WebException

Question

I'm working on resolving this bug:
https://github.com/openstacknetsdk/openstack.net/issues/333

The issue involves a ProtocolViolationException with the following message:

Chunked encoding upload is not supported on the HTTP/1.0 protocol.

I found that I am able to reliably reproduce the issue my making a web request that produces a 502 response code, followed by the call to use a POST request with chunked encoding. I traced this back to the ServicePoint.HttpBehaviour property having the value HttpBehaviour.HTTP10 following the 502 response.

I was able to resolve the issue using the following hack (in the catch block). This code "hides" the ServicePoint instance created by the failed request from the ServicePointManager, forcing it to create a new ServicePoint for the next request.

public void TestProtocolViolation()
{
    try
    {
        TestTempUrlWithSpecialCharactersInObjectName();
    }
    catch (WebException ex)
    {
        ServicePoint servicePoint = ServicePointManager.FindServicePoint(ex.Response.ResponseUri);
        FieldInfo table = typeof(ServicePointManager).GetField("s_ServicePointTable", BindingFlags.Static | BindingFlags.NonPublic);
        WeakReference weakReference = (WeakReference)((Hashtable)table.GetValue(null))[servicePoint.Address.GetLeftPart(UriPartial.Authority)];
        if (weakReference != null)
            weakReference.Target = null;
    }

    TestTempUrlExpired();
}

Questions:

1. Why am I observing this behavior?
2. What is a non-hacky way to resolve the issue?

Scowl

Answer 1

Q. Why am I observing this behavior?

A. The .NET framework's support for connections to HTTP servers is based on ServicePointManager providing ServicePoint instances. Each ServicePoint instance assumes it is connecting to a single "logical" service based on the endpoint address. This object caches certain information about the service on the other end, and one of those pieces of information is whether or not the service supports HTTP/1.1. If any request to the service indicates that the service only supports HTTP/1.0, the ServicePoint "latches" into that state, and the ServicePointManager will only recreate a fresh ServicePoint not in that state if/when the garbage collector clears the WeakReference pointing to the instance.

This behavior was likely deemed to not be a problem for the following reasons:

1. Typically, a single endpoint is served by a single service, and that service either does or does not support HTTP/1.1.

2. If an endpoint is actually a load balancer that dispatches requests to multiple backing HTTP implementations (generally across multiple nodes), those nodes represent multiple instances of the same overall service installation, and either all nodes support HTTP/1.1 or none of the nodes does.

3. In the rare case that the above do not hold, the lack of HTTP/1.0 features is generally not an impediment to services. An endpoint deploying one or more HTTP/1.0 servers is unlikely to require clients to send requests using HTTP/1.1 features.

Q. Is there a non-hacky way to resolve the issue?

A. There are certainly work-arounds, but one or more of the options may be unsuitable for the particular environment. The following lists a few of these options.

1. Update the service to meet the conditions listed above. If you are providing the service that does not meet the above conditions, you should consider updating the service based on the understanding that .NET clients may not be able to communicate with your service under some scenarios. If you do not have control over the service, the obviously this is not a viable solution.

2. Consider alternatives to using chunked encoding for uploading files. If you know the size of your stream, you may not need to use chunked encoding, which avoids the dependency on HTTP/1.1. For the case of the SDK mentioned in the question, the underlying SimpleRESTServices library actually requires the stream size to be known in advance, so chunked encoding is not actually being used for its intended purpose. Instead, the library should use buffered transfers when the content length is known in advance, and only rely on chunked encoding when the Stream.Size property throws a NotSupportedException.

3. Consider setting HttpWebRequest.AllowWriteStreamBuffering to true. While I have not tested this solution, information gathered while browsing the reference source suggests that this property allows the implementation to fall back to buffering in the case where chunked transfers are not supported, rather than simply throwing the ProtocolViolationException.

4. Force the ServicePoint to time out my setting ServicePoint.MaxIdleTime to 0. This is still hacky, but doesn't rely on reflection and should still work on Mono. The modified code would look like the following.

   public void TestProtocolViolation()
   {
       try
       {
           TestTempUrlWithSpecialCharactersInObjectName();
       }
       catch (WebException ex)
       {
           ServicePoint servicePoint = ServicePointManager.FindServicePoint(ex.Response.ResponseUri);
           if (servicePoint.ProtocolVersion < HttpVersion.Version11)
           {
               int maxIdleTime = servicePoint.MaxIdleTime;
               servicePoint.MaxIdleTime = 0;
               Thread.Sleep(1);
               servicePoint.MaxIdleTime = maxIdleTime;
           }
       }
   
       TestTempUrlExpired();
   }
   

Scowl

Has your pull request to fix this been accepted into openstack? – Pasteurism

My application is experiencing this same issue, but the issue it's causing it slightly different. What we discovered is that once we "latch" as you put it to Http 1.0 it disables keep-alive since it's not longer the default and it also disables pipelining. Because we're communicating via https to a set of servers in the AWS cloud where we have HAProxies terminating the SSL, we are overwhelming the HAProxies and timeouts and slow responses occur. – Jessen