How to mix compression and caching in HttpWebRequest?

Question

I have a c# client talking to a cherrypy(http/rest) webservice. The problem is i can't both turn on compression and caching at the same time.

request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;

By leaving out the above line I get the correct caching headers (If-None-Math,If-Modified-Since) while commenting it out gets me the compression headers (Accept-Encodig:gzip) but not the caching headers. It seems to me like a bug but maybe i'm doing something wrong.

[full code]

        public static string GET(string URL)
    {
        string JSON;
        // Create the web request  
        HttpWebRequest request = WebRequest.Create(URL) as HttpWebRequest;
        HttpRequestCachePolicy cPolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.Revalidate);

        request.Accept = "application/json";            
        request.CachePolicy = cPolicy;
        request.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
        request.Pipelined = false;

        // Get response  
        using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
        {                
            // Get the response stream  
            StreamReader readerF = new StreamReader(response.GetResponseStream());

            JSON = readerF.ReadToEnd();
            // Console application output  
            //Console.WriteLine(JSON);
            if (response.IsFromCache )
                Console.WriteLine("Request not from cache");
        }

        return JSON;
    }

Stiffnecked

Answer 1

I have implemented a workaround, see code below. I judged handling the compression easier than handling the cacheing so I implemented the compression part myself. Quite easy thanks to a blog post: HttpWebRequest and GZip Http Responses; I still think this is a bug in .net.

        public static string GET(string URL)
    {
        string JSON;
        // Create the web request  
        HttpWebRequest request = WebRequest.Create(URL) as HttpWebRequest;
        HttpRequestCachePolicy cPolicy = new HttpRequestCachePolicy(HttpRequestCacheLevel.Revalidate);

        request.Accept = "application/json";
        request.Headers.Add(HttpRequestHeader.AcceptEncoding, "gzip,deflate");
        request.CachePolicy = cPolicy;
        request.Pipelined = false;

        // Get response  
        using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
        {
            //From http://www.west-wind.com/WebLog/posts/102969.aspx
            Stream responseStream = responseStream = response.GetResponseStream();
            if (response.ContentEncoding.ToLower().Contains("gzip"))
                responseStream = new GZipStream(responseStream, CompressionMode.Decompress);
            else if (response.ContentEncoding.ToLower().Contains("deflate"))
                responseStream = new DeflateStream(responseStream, CompressionMode.Decompress);


            // Get the response stream  
            StreamReader readerF = new StreamReader(responseStream);
            JSON = readerF.ReadToEnd();
        }

        return JSON;
    }

Stiffnecked

Answer 2

Is this a side effect of the policy? What happens if you just use the default policy, or other policies?

Other option is to manage the cache yourself.

Quartziferous

I tried your suggestion, setting the cache policy to default. Same results, seems like setting compression disables policy. I know i can implement a cache myself, but that is not exaclty trivial :-) – Stiffnecked