Why does HttpWebRequest throw an exception instead returning HttpStatusCode.NotFound?

Question

I'm trying to verify the existence of a Url using HttpWebRequest. I found a few examples that do basically this:

HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(Url);
request.Method = "HEAD";
using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
{
    return response.StatusCode;
}

However, if the url is indeed broken, it's not returning a response, it's instead throwing an exception.

I modified my code to this:

try
{
    HttpWebRequest request = (HttpWebRequest)HttpWebRequest.Create(Url);
    request.Method = "HEAD";
    using (HttpWebResponse response = request.GetResponse() as HttpWebResponse)
    {
        return response.StatusCode;
    }
}
catch (System.Net.WebException ex)
{
    var response = ex.Response as HttpWebResponse;
    return response == null ? HttpStatusCode.InternalServerError : response.StatusCode;
}

which seems to finally do what I want.

But I would like to know, why is the request throwing an exception instead of returning the response with a NotFound status code?

Lajoie

Answer 1

Ya this can be quite annoying when web pages use status codes heavily and not all of them are errors. Which can make processing the body quite a pain. Personally I use this extension method for getting the response.

public static class HttpWebResponseExt
{
    public static HttpWebResponse GetResponseNoException(this HttpWebRequest req)
    {
        try
        {
            return (HttpWebResponse)req.GetResponse();
        }
        catch (WebException we)
        {
            var resp = we.Response as HttpWebResponse;
            if (resp == null)
                throw;
            return resp;
        }
    }
}

Arise

While this is the least work to save code using HttpWebRequest/Response from this bad design choice on the .Net Framework authors' part, the correct solution is to use HttpClient, which does not throw on 4xx and 5xx status codes. Exceptions are meant for exceptional situations, and throwing just to catch it and proceed as though it's fine like this is ugly and bad for performance, especially given there's a better option that avoids it entirely. msdn.microsoft.com/en-us/library/hh138242(v=vs.118).aspx – Gunfight

this does not appear to be true; I'm using HttpClient in a project and when calling a url that does not exist which returns a 404 status code, the client is throwing an exception instead of returning a response with the 404 status code. is there an extra step in using httpclient to prevent this? – Lajoie

In my case HttpClient throws exception on 400 Bad Request. – Abele

In my case an AggregateException was thrown and the InnerException was a WebException. – Gules

This article also reflects the solution above, but may add clarity: techbrij.com/handle-web-exception-raised-httpwebrequest-csharp – Misty

Answer 2

Why not? They're both valid design options, and HttpWebRequest was just designed to work this way.

Accalia

As long as you can read the response headers and body when the code is 4xx – Vevina

I guess I was confused, because none of the code samples I saw for this took this into consideration. many didn't even have a try/catch, and I was wondering if perhaps I missed something and there is a way to get the status without throwing an exception. it just seems counterintuitive to throw an entire exception if the statuscode is designed to handle such a status – Lajoie

Yeah, it always leads to interesting results when you think that "of course xxx tested the line of code he put on his website!" and you turn out to be mistaken :) – Accalia

But Microsoft recommends to avoid exception where possible as it can negatively impact performance. msdn.microsoft.com/en-us/library/ms229009.aspx – Mechanic

this is another reason I believe this to be unintuitive. if the response can capture 404 as a status code then I don't think that should be considered an exception. the request completed, it just didn't find anything. – Lajoie

@MahmoudAl-Qudsi, I'm glad they didn't decide to return exception for 3xx and 20x too, just because you can catch them later! 404 is not an exception and clearly is not an exception on client side, but server side which is another machine which need to handle the exception. and exception on my side means I did something wrong! – Miscue

Answer 3

Just like @Will, I wrote similar extension method to get the response content in string from WebException.

        /// <summary>
        /// Reads Response content in string from WebException
        /// </summary>
        /// <param name="webException"></param>
        /// <returns></returns>
        public static (HttpStatusCode statusCode, string? responseString) GetResponseStringNoException(this WebException webException)
        {
            if (webException.Response is HttpWebResponse response)
            {
                Stream responseStream = response.GetResponseStream();

                StreamReader streamReader = new(responseStream, Encoding.Default);

                string responseContent = streamReader.ReadToEnd();
                HttpStatusCode statusCode = response.StatusCode;

                streamReader.Close();
                responseStream.Close();
                response.Close();

                return (statusCode, responseContent);
            }
            else
            {
                return (HttpStatusCode.InternalServerError, null);
            }
        }

The above code is non-optimised solution.

Sulphate