Tracking progress of HttpWebRequest and HttpWebResponse
Asked Answered
H

5

7

I'm using Sharefile API that sends HTTP requests and gets their respective answers. They are made through URL's and use always the same function. Here it is.

Private Function InvokeShareFileOperation(ByVal requestUrl As String) As JObject

    Dim request As HttpWebRequest = WebRequest.Create(requestUrl)
    Dim response As HttpWebResponse = request.GetResponse()

    Dim reader As StreamReader = New StreamReader(response.GetResponseStream())

    Dim json As String = reader.ReadToEnd()
    response.Close()
    Return JObject.Parse(json)

End Function

As some operations are a bit long, I need to somehow track their progress while they are underway and don't know how to do it. Later I'm intending to use that progress and create a progress bar from it.

(EDIT) By the way, it's the second code line (below) that takes most time, that is the operation to track.

Dim response As HttpWebResponse = request.GetResponse()
Hight answered 8/1, 2014 at 10:47 Comment(7)
You should put the HttpWebResponse and StreamReader` into Using blocks, especially since you use them frequently. This could be part of the slowdown.Tatty
What do you think about my answer?Pigment
what size is the JSON file returned from the API usually? If it is only small, less than 1k for instance, then it will be read from the response stream in one single read. This would make a progress bar pointless as there's only one measurement, and getting the progress of GetResponse() is impossible. In your reply to @Pigment it seemed like the content length was only 50 bytesKaden
Yeah, I'm guessing it's impossible too. There should be a way to get the progress somehow but unfortunately, I don't think there isHight
If its just the first API request that takes ages and then the following ones are pretty quick then this could have something to do with your proxy settingsKaden
That's not the case, API requests are made anytime and some are longer than others. Again, the issue here isn't the time elapsed but the tracking of itHight
I can't make it work. Anyway, it would show progress for downloading body, and in your case it's done in way less then second.Pigment
K
5

EDIT: I don't think you are going to be able to measure progress in any accurate way here as the bulk of the operation seems to be reliant on the remote server processing the request. GetResponse() handles setting up the DNS, connecting, sending and waiting for the remote server and this is all out of your hands. Reading the response stream is only measurable if the content-length header is returned. Personally I would show progress as 20% initially, 60% when GetResponse returns, and then the last 40% could be shown incrementally if you have the content length before downloading, or done in one go once you have finished reading the response stream.

As its a web request you can find out the content length first and then read the response stream using a buffer instead of ReadToEnd(). This allows you to calculate the progress and fire off notifications while downloading the response.

Dim request As HttpWebRequest = WebRequest.Create(requestUrl)

Using response As HttpWebResponse = request.GetResponse()
  Dim contentLength As Long = response.ContentLength
  Dim bytesReceived As Long
  Dim bufferLength as Integer = 8192
  Dim buffer(bufferLength) As Char
  Dim sb As New StringBuilder

  Using reader As StreamReader = New StreamReader(response.GetResponseStream())
    Do
      Dim bufferedCount As Integer = reader.Read(buffer, 0, bufferLength)
      sb.Append(buffer, 0, bufferedCount)
      bytesReceived += bufferedCount
      Console.WriteLine(bytesReceived / contentLength * 100 & "%")
    Loop While bytesReceived < contentLength
  End Using

  Return JObject.Parse(sb.ToString)
End Using

Obviously you can substitute the Console.WriteLine with a progress update function or a call to a SignalR hub to update a web page, and you can experiment with the buffer size to see what works best for you.

Kaden answered 13/1, 2014 at 0:26 Comment(4)
Thanks for your answer. The loop is an infinite cycle, bytesReceived is always < contentLength and therefore never stops. Anyway, you get the contentLength after the getResponse, meaning, after the operation is done. Even if I can manipulate that information later, the operation is already done and the time i took too.Hight
GetResponse() should only get the headers - the actual content is downloaded using the response stream. There is no way to measure the progress of the remote server processing the request until after GetResponse returns. Fetching response.ContentLength may be forcing the full download to occur though - can you try response.GetResponseHeader("content-length") insteadKaden
It would be helpful if you can provide the actual response headers from the remote server - maybe using Fiddler or by making the request through a browser and capture it using the browser dev tools. If the response is sent chunked then I don't think the content length header is set. Getting the content length is the key thing here:)Kaden
You could also try inserting request.Proxy = Nothing at the top before the Using response line. I remember having big delays on an older .net version and this solved it. In fact this could be your problem and I really hope it is because its such an easy fix:)Kaden
P
3

First we must find out what's slowing down. Request isn't send until GetResponse() is called, so processing by server can take some time. Downloading can also take some time. If response is small (relative to connection speed), you can't do much (you can if server is yours, but we'll focus on client) because you can't get progress from server. If response is large, and you want to track downloading, you can only do it if you have Content-Length header. And to get only headers, server must support HEAD request method. So here is code :

Imports System
Imports System.Net
Imports System.IO
Imports System.Text
Imports System.Threading
Imports Microsoft.VisualBasic

Public Class Form1

    Private Function InvokeShareFileOperation(ByVal requestUrl As String) As JObject
        HTTPWebRequest_GetResponse.Main(requestUrl)
        ProgressBar1.Value = 0
        Dim result As String
        Do
            Try
                ProgressBar1.Value = HTTPWebRequest_GetResponse.progress
            Catch ex As ArgumentOutOfRangeException
                ProgressBar1.Style = ProgressBarStyle.Marquee
            End Try
            If HTTPWebRequest_GetResponse.done = True Then
                result = HTTPWebRequest_GetResponse.response
                ProgressBar1.Style = ProgressBarStyle.Continuous
                ProgressBar1.Value=100
                Debug.WriteLine(result)
                Return JObject.Parse(result)
                Exit Do
            End If
        Loop
    End Function

End Class


Public Class RequestState
    ' This class stores the State of the request. 
    Private BUFFER_SIZE As Integer = 1024
    Public requestData As StringBuilder
    Public BufferRead() As Byte
    Public request As HttpWebRequest
    Public response As HttpWebResponse
    Public streamResponse As Stream

    Public Sub New()
        BufferRead = New Byte(BUFFER_SIZE) {}
        requestData = New StringBuilder("")
        request = Nothing
        streamResponse = Nothing
    End Sub 'New 
End Class 'RequestState


Class HTTPWebRequest_GetResponse

    Private BUFFER_SIZE As Integer = 1024
    Public Shared response As String
    Public Shared done As Boolean = False
    Public Shared length As Long = 1
    Public Shared progress As Integer
    Public Shared myHttpWebRequest As HttpWebRequest
    Public Shared myRequestState As New RequestState()

    Shared Sub Main(url As String)

        Try
            Dim headRequest As HttpWebRequest = WebRequest.Create(url)
            headRequest.Method = "HEAD"
            Dim headResponse As HttpWebResponse = headRequest.GetResponse
            length = headResponse.ContentLength
            Debug.WriteLine(length)
            headResponse.Close()
            ' Create a HttpWebrequest object to the desired URL.  
            myHttpWebRequest = WebRequest.Create(url)

            ' Create an instance of the RequestState and assign the previous myHttpWebRequest 
            ' object to its request field.   

            myRequestState.request = myHttpWebRequest
            'Dim myResponse As New HTTPWebRequest_GetResponse()

            ' Start the asynchronous request. 
            Dim result As IAsyncResult = CType(myHttpWebRequest.BeginGetResponse(New AsyncCallback(AddressOf RespCallback), myRequestState), IAsyncResult)

        Catch e As WebException
            Debug.WriteLine("Main Exception raised!")
            Debug.WriteLine("Message: " + e.Message)
            Debug.WriteLine("Status: " + e.Status)
        Catch e As Exception
            Debug.WriteLine("Main Exception raised!")
            Debug.WriteLine("Source : " + e.Source)
            Debug.WriteLine("Message : " + e.Message)
        End Try
    End Sub 'Main

    Private Shared Sub RespCallback(asynchronousResult As IAsyncResult)
        Debug.WriteLine("RespCallBack entered")
        Try
            ' State of request is asynchronous. 
            Dim myRequestState As RequestState = CType(asynchronousResult.AsyncState, RequestState)
            Dim myHttpWebRequest As HttpWebRequest = myRequestState.request
            myRequestState.response = CType(myHttpWebRequest.EndGetResponse(asynchronousResult), HttpWebResponse)

            ' Read the response into a Stream object. 
            Dim responseStream As Stream = myRequestState.response.GetResponseStream()
            myRequestState.streamResponse = responseStream

            ' Begin the Reading of the contents of the HTML page. 
            Dim asynchronousInputRead As IAsyncResult = responseStream.BeginRead(myRequestState.BufferRead, 0, 1024, New AsyncCallback(AddressOf ReadCallBack), myRequestState)
            Return
        Catch e As WebException
            Debug.WriteLine("RespCallback Exception raised!")
            Debug.WriteLine("Message: " + e.Message)
            Debug.WriteLine("Status: " + e.Status)
        Catch e As Exception
            Debug.WriteLine("RespCallback Exception raised!")
            Debug.WriteLine("Source : " + e.Source)
            Debug.WriteLine("Message : " + e.Message)
        End Try
    End Sub 'RespCallback

    Private Shared Sub ReadCallBack(asyncResult As IAsyncResult)
        Debug.WriteLine("ReadCallBack entered")
        Try

            Dim myRequestState As RequestState = CType(asyncResult.AsyncState, RequestState)
            Dim responseStream As Stream = myRequestState.streamResponse
            Dim read As Integer = responseStream.EndRead(asyncResult)
            ' Read the HTML page. 
            If read > 0 Then
                myRequestState.requestData.Append(Encoding.ASCII.GetString(myRequestState.BufferRead, 0, read))
                If length = -1 Or length = 0 Then
                    progress = -1
                Else
                    progress = myRequestState.BufferRead.Length * 100 / length
                    Debug.WriteLine(progress)
                End If
                Dim asynchronousResult As IAsyncResult = responseStream.BeginRead(myRequestState.BufferRead, 0, 1024, New AsyncCallback(AddressOf ReadCallBack), myRequestState)

            Else
                If myRequestState.BufferRead.Length > 1 Then
                    Dim fullResponse As String = myRequestState.requestData.ToString
                    response = fullResponse.Substring(0, fullResponse.IndexOf("</body>")).Substring(fullResponse.IndexOf(">", fullResponse.IndexOf("<body")) + 2) 'Returns only body
                    ' Release the HttpWebResponse resource.
                    myRequestState.response.Close()
                    done = True
                    Debug.WriteLine(done)
                End If

                responseStream.Close()
            End If

        Catch e As WebException
            Debug.WriteLine("ReadCallBack Exception raised!")
            Debug.WriteLine("Message: " + e.Message)
            Debug.WriteLine("Status: " + e.Status)
        Catch e As Exception
            Debug.WriteLine("ReadCallBack Exception raised!")
            Debug.WriteLine("Source : " + e.Source)
            Debug.WriteLine("Message : " + e.Message)
        End Try
    End Sub 'ReadCallBack 
End Class 'HttpWebRequest_BeginGetResponse

I took code from http://msdn.microsoft.com/en-us/library/debx8sh9(v=vs.110).aspx and changed it.

EDIT: Code now returns only body and response is closed.

EDIT2: As @Geezer68 said, it's not 100% accurate, but it's OK for showing progress to user.

Pigment answered 13/1, 2014 at 11:19 Comment(13)
I'm sorry @srka, your solution doesn't work, it causes an infinite loopHight
@Hight It works fine for me. Where is infinte loop?Pigment
@Hight I found issue with setting ProgressBar value and edited answer. Maybe that made you think there's an infininite loop?Pigment
Infinite loop again: Inside the "InvokeSharefileOperation" function the line "ProgressBar1.Value = HTTPWebRequest_GetResponse.progress" stops at a value. That way, it keeps looping and setting the progressBar value as always the same. It never exits the "do" cycle as the "getResponse.done is never" true. Did you test it yourself? I ask that because some "imports" were missingHight
Just been thinking that if the API being called is taking ages to come back with the initial response then making a HEAD and a GET request is just going to make things take a lot longer - it might not even allow the HEAD request. But the async method of reading the response that you've given is the much better wayKaden
@Kaden I agree, but this is only way I know.Pigment
@Hight That's weird. I tested it and it worked fine. I added few Debug.WriteLine()s so send me output.Pigment
50; RespCallBack entered; ReadCallBack entered; 2050; ReadCallBack entered; A first chance exception of type 'System.ArgumentOutOfRangeException' occurred in mscorlib.dll; ReadCallBack Exception raised!; Source : mscorlib; Message : Length cannot be less than zero.; Parameter name: length;Hight
it sticks with 2050 as valueHight
@Hight What line is throwing that exception?Pigment
when it goes through the ProgressBar1.Value = HTTPWebRequest_GetResponse.progress for the second timeHight
@Hight My only guess for what's causing that exception is that Content-Length is body's length, but in calculating progress we use whole file's length, so progress is larger than 100. I will change code to calculate progress after <body> part, but i'll have time in about 10 hours.Pigment
let us continue this discussion in chatHight
S
2

I'm pretty sure what you want is reader.BaseStream.Length so you can know the length before reading. (At least I did, so I tried) But it threw a NotSupportedException with the message This stream does not support seek operations. So I googled StreamReader + This stream... and found this SO link:

Error “This stream does not support seek operations” in C#

So the short answer is: It is not possible.

Sarchet answered 12/1, 2014 at 20:16 Comment(3)
OK, it is not possible. Is there a way around it? To achieve the same result?Hight
@Hight I'm yet to find a solution. I've searched and searched. I even tried the WebClient, but it threw a NotSupportedException, it doesn't support seeking. So unless the content-length is available, I think it's a lost cause. I'm sorry.Behindhand
Thanks for your efforts man, it really is tricky. I haven't found a solution myself despite not searching for anything new since last Friday. I'll be trying some more, maybe tomorrow I'll dig into it again.Hight
R
1

Maybe a simple stopwatch is a way to start with ?

    Dim timer As System.Diagnostics.Stopwatch = New Stopwatch()

    Dim request As HttpWebRequest = WebRequest.Create(requestUrl)

    timer.Start()
    Dim response As HttpWebResponse = request.GetResponse()
    timer.Stop()

    Dim reader As StreamReader = New StreamReader(response.GetResponseStream())
    Dim json As String = reader.ReadToEnd()
    response.Close()

    Label1.Text = "Secs:" & timer.Elapsed.ToString()
Risk answered 8/1, 2014 at 21:23 Comment(1)
Thank you @carleson. Well, your suggestion is good: I get the time elapsed indeed. But I can't relate that to the progress of the operation. I don't know if it's gonna last 5 or 10 secs (it's variable) and so I can't tell if at 4 secs I'm at 80% or at 40% of the operation.Hight
P
0

Here is a link to Microsoft Example where you set the buffer size and a call back on the response object https://msdn.microsoft.com/en-us/library/86wf6409%28v=vs.110%29.aspx

Plasmosome answered 30/4, 2015 at 6:49 Comment(1)
“While this link may answer the question, it is better to include the essential parts of the answer here and provide the link for reference. Link-only answers can become invalid if the linked page changes.”Salvador

© 2022 - 2024 — McMap. All rights reserved.