How can I read the response from a web request when the Status is not 200?

Question

I am having difficulty getting the response text from a HTTP web request in vb.net when I get a web exception.

This is the code I am doing it with.

Try
            myWebResponse = CType(request.GetResponse(), HttpWebResponse)
            myStreamReader = New StreamReader(myWebResponse.GetResponseStream())

            ResponseText = myStreamReader.ReadToEnd
            If myWebResponse.StatusCode = HttpStatusCode.Accepted Or myWebResponse.StatusCode = 200 Then
                SendResult = True 'Sent 
                SendStatus = 1 'message sent successfully
                Try
                    Integer.TryParse(myWebResponse.Headers("Number-Of-MT-PDU"), num_MT_PDU)
                Catch ex As Exception
                End Try
            Else
                SendStatus = 2 'message processed but not sent successfully
            End If
        Catch e As WebException
            If (e.Status = WebExceptionStatus.ProtocolError) Then
                Dim response As WebResponse = e.Response
                Using (response)
                    Dim httpResponse As HttpWebResponse = CType(response, HttpWebResponse)
                    statusCode = httpResponse.StatusCode
                    Try
                        myStreamReader = New StreamReader(response.GetResponseStream())
                        Using (myStreamReader)
                            ResponseText = myStreamReader.ReadToEnd & "Status Description = " & HttpWebResponse.StatusDescription
                        End Using
                    Catch ex As Exception
                        Logger.LogError(Me, ex)
                    End Try
                End Using

Annoyingly, the API I am contacting uses a 404 as a valid response. If I put the request in a browser some message text will be displayed. I want to be able to use that text in my program. I can not simply use the error code to determine actions as I don't think I can differentiate between a valid 404 response and an actual error.

In the code this line

myWebResponse = CType(request.GetResponse(), HttpWebResponse)

throws an exception.

In the exception I can get the 404 code and the description but not the response stream. It is always null.

If I get a 200 response I get the text in the Response stream no problem.

In the web exception response object (in Visual Studios debugger) I have checked the headers and the object values and can't find the response text anywhere. If I stick the request URL in a browser I get response text back even though it is a 404.

The raw response in fiddler:

HTTP/1.1 404 Not Found Connection: close Content-Type: text/plain; charset=UTF-8 Content-Length: 35 "The response Message"

Any ideas on how I can get "The response Message" in my program? I have to use .Net on the server.

Thanks for any help anybody can give.

Slue

One thing that might provide a clue would be to use Fiddler to see what is actually coming back over the wire – Sair

I think I had the same problem before. Can't use webclient. Hang on, let me look that up. – Damp

I asked similar question #9062589. No answer there either. – Damp

I just hit localhost/test (which doesn't exist) using WebClient.DownloadData and the base WebException's (I'm using DotLisp, so there's a BacktraceException and a TargetInvocationException "on top".) Response exists and the equivalent of New StreamReader(response.GetResponseStream()).ReadToEnd works fine. – Imogen

Answer 1

This LINQPad query works fine, dumping the HTML provided by my web server's "Not Found" error web page:

Dim rq = System.Net.WebRequest.Create(New Uri("http://localhost/test"))
Try 
  Dim rs = rq.GetResponse
  rs.Dump
Catch Ex As System.Net.WebException
 Dim rs = Ex.Response
 Call (New StreamReader(rs.GetResponseStream)).ReadToEnd.Dump
End Try

FYI Your code works for me, except the presumed typo re HttpWebResponse.StatusDescription (and commenting out "unrelated stuff"), again as a LINQPad query (in .NET 4.0):

Dim request = WebRequest.Create("http://localhost/test")
Dim myStreamReader As StreamReader
Dim SendStatus As Integer = -1
Dim statusCode As HttpStatusCode
Dim ResponseText As String
Try
    Dim myWebResponse = CType(request.GetResponse(), HttpWebResponse)
        myStreamReader = New StreamReader(myWebResponse.GetResponseStream())

        ResponseText = myStreamReader.ReadToEnd
        If myWebResponse.StatusCode = HttpStatusCode.Accepted Or myWebResponse.StatusCode = 200 Then
            'SendResult = True 'Sent 
            SendStatus = 1 'message sent successfully
            'Try
            '    Integer.TryParse(myWebResponse.Headers("Number-Of-MT-PDU"), num_MT_PDU)
            'Catch ex As Exception
            'End Try
        Else
            SendStatus = 2 'message processed but not sent successfully
        End If
    Catch e As WebException
        If (e.Status = WebExceptionStatus.ProtocolError) Then
            Dim response As WebResponse = e.Response
            Using (response)
                Dim httpResponse As HttpWebResponse = CType(response, HttpWebResponse)
                statusCode = httpResponse.StatusCode
                Try
                    myStreamReader = New StreamReader(response.GetResponseStream())
                    Using (myStreamReader)
                        ResponseText = myStreamReader.ReadToEnd & "Status Description = " & httpResponse.StatusDescription ' HttpWebResponse.StatusDescription
                    End Using
                Catch ex As Exception
                    'Logger.LogError(Me, ex)
                    ex.Dump("Exception")
                End Try
            End Using
      End If
End Try
ResponseText.Dump("ResponseText")

I have also confirmed the above code (with the inferred As clauses added and converting the .Dump calls to Console.WriteLine) works in .NET 2.0 with VB8.

Imogen

Hi thanks a lot for your response. It's been a while since I asked the question and I don't have access to the original code or server where the issue occurred but I will accept your answer as it looks fine. The only thing is I was stuck on a server running .Net 2.0 (company was obviously afraid of change) and I have some memory of being able to get standard 404 server response text but not the text the API I was calling returned. Anyway, hopefully this will all help someone, somewhere. – Ecumenical

Thanks this answer was very helpful – Digest

Answer 2

Note that the key is that even though the act of GetResponseStream() throws a .NET WebException, the HttpWebResponse is actually passed to the WebException object, so when in the Catch, you do a new GetResponseStream() on the WebException.Response object.

Below, very similar code for when in the Catch of the initial GetResponseStream()

   Try 
            OriginalResponseStream = GetResponseStream(OriginalHTTPWebResponse)

   Catch wex as WebException
            Dim response As WebResponse = wex.Response
            Dim statusCode As HttpStatusCode
            Dim ResponseText As String

            Dim httpResponse As HttpWebResponse = CType(response, HttpWebResponse)
            statusCode = httpResponse.StatusCode

            Try

                Dim myStreamReader As New StreamReader(response.GetResponseStream())
                Using (myStreamReader)
                    ResponseText = myStreamReader.ReadToEnd
                    Process(ResponseText) '<===as in whatever you need to do with the response
                End Using
            Catch ex As Exception

                HandleIt(ex.Message) '<===as in whatever you want to do if Exception during the above

            End Try

   End Try

Khedive