Why does sending post data with WebRequest take so long?

Question

I am currently creating a C# application to tie into a php / MySQL online system. The application needs to send post data to scripts and get the response.

When I send the following data

username=test&password=test  

I get the following responses...

Starting request at 22/04/2010 12:15:42  
Finished creating request : took 00:00:00.0570057  
Transmitting data at 22/04/2010 12:15:42  
Transmitted the data : took 00:00:06.9316931       <<--
Getting the response at 22/04/2010 12:15:49  
Getting response 00:00:00.0360036  
Finished response 00:00:00.0360036  
Entire call took 00:00:07.0247024  

As you can see it is taking 6 seconds to actually send the data to the script, I have done further testing bye sending data from telnet and by sending post data from a local file to the url and they dont even take a second so this is not a problem with the hosted script on the site.

Why is it taking 6 seconds to transmit the data when it is two simple strings?

I use a custom class to send the data

class httppostdata
{
    WebRequest request;
    WebResponse response;

    public string senddata(string url, string postdata)
    {
        var start = DateTime.Now;
        Console.WriteLine("Starting request at " + start.ToString());

        // create the request to the url passed in the paramaters
        request = (WebRequest)WebRequest.Create(url);


        // set the method to post
        request.Method = "POST";
        // set the content type and the content length
        request.ContentType = "application/x-www-form-urlencoded";
        request.ContentLength = postdata.Length;
        // convert the post data into a byte array
        byte[] byteData = Encoding.UTF8.GetBytes(postdata);
        var end1 = DateTime.Now;
        Console.WriteLine("Finished creating request : took " + (end1 - start));

        var start2 = DateTime.Now;
        Console.WriteLine("Transmitting data at " + start2.ToString());
        // get the request stream and write the data to it
        Stream dataStream = request.GetRequestStream();
        dataStream.Write(byteData, 0, byteData.Length);
        dataStream.Close();
        var end2 = DateTime.Now;
        Console.WriteLine("Transmitted the data : took " + (end2 - start2));


        // get the response
        var start3 = DateTime.Now;
        Console.WriteLine("Getting the response at " + start3.ToString());


        response = request.GetResponse();
        //Console.WriteLine(((WebResponse)response).StatusDescription);
        dataStream = response.GetResponseStream();
        StreamReader reader = new StreamReader(dataStream);
        var end3 = DateTime.Now;
        Console.WriteLine("Getting response " + (end3 - start3));

        // read the response
        string serverresponse = reader.ReadToEnd();
        var end3a = DateTime.Now;
        Console.WriteLine("Finished response " + (end3a - start3));

        Console.WriteLine("Entire call took " + (end3a - start));

        //Console.WriteLine(serverresponse);
        reader.Close();
        dataStream.Close();
        response.Close();

        return serverresponse;
    }
}

And to call it I use

private void btnLogin_Click(object sender, EventArgs e)
{
    // string postdata;

    if (txtUsername.Text.Length < 3 || txtPassword.Text.Length < 3)
    {
        MessageBox.Show("Missing your username or password.");
    }
    else
    {
        string postdata = "username=" + txtUsername.Text +
                          "&password=" + txtPassword.Text;

        httppostdata myPost = new httppostdata();
        string response = myPost.senddata("http://www.domainname.com/scriptname.php", postdata);
        MessageBox.Show(response);
    }
}

Bluegreen

I realize it's http in your example, but in your actual six-second test scenario is the URL https? Could be overhead due to handshake, checking CRL, calling OCSP, etc. – Kraut

Answer 1

Make sure you explicitly set the proxy property of the WebRequest to null or it will try to autodetect the proxy settings which can take some time.

Ashlan

If its the case of proxies would I be better do a splash screen load and doing a blank call? the reason I ask is because the software is being used on multiple networks from laptops and some networks use proxies – Bluegreen

Answer 2

Chances are that because, in your test, you only call this once, the delay you see is the C# code being JIT compiled.

A better test would be to call this twice, and discard the timings from the first time and see if they are better.

An even better test would be to discard the first set of timings, and then run this many times and take an average, although for a very loose "indicative" view, this is probably not necessary.

As an aside, for this sort of timing, you are better off using the System.Diagnostics.Stopwatch class over System.DateTime.

[EDIT] Also, noting Mant101's suggestion about proxies, if the setting no proxy fails to resolve things, you may wish to set up Fiddler and set your request to use Fiddler as its proxy. This would allow you to intercept the actual http calls so you can get a better breakdown of the http call timings themselves from outside the framework.

Fanchet

This is so wrong. (1) The method is JIT compiled before it is run, not halfway through it. (2) JIT compiling such a short method does not take 6 seconds. (3) The time it takes to get the timings is insignificant compared to 6 seconds. (4) You take averages when you're measuring short, time-critical operations that take milliseconds; not when communicating with an HTTP server takes 6 seconds. (5) While the accuracy of DateTime.Now is not as accurate as a Stopwatch, in the range of seconds the difference in accuracy is totally insignificant. – Gatling

1) THIS method may be jitted before it is run, but you will notice it calls other methods. It is these that may be jitted as this method progresses. 2) The time it takes to JIT something is dependent on many things including current memory use (and the need to page out), disk speed, etc. Generally it is fast. Always? no. 3) I never said it wasn't 4) My point being - that if this slow experience was due to something like disk IO, averging may help reduce the impact. I was not talking about averaging to remove timing jitter. Your assumption, not mine. 5) I said "as a general rule". – Fanchet

Addendum to your point 4 - it is implicit in your phrasing that you feel the 6 seconds is due to communicating with the HTTP server. I am suggesting it may not be. You do average short-timecritical operations to reduce the impact of clock jitter and other random error sources. You also may do the same, to reduce the impact, on a computer, of your machine being busy performing other tasks that are not obvious at test time. It may well be worth running the same task again and again, possibly at different times, in order to eliminate impact of hidden scheduled or other tasks. – Fanchet

Thanks for the stopwatch recommendation i'll look into it, as for the JIT it isn't that as I have done several calls to the same function one after the other and always get 5+ seconds and the system here is a fully fledged server with 6Gb ram built soley for development – Bluegreen

Yes - in that case you probably have eliminated the JITing as a cause. – Fanchet

it would appear to be proxy, but how can I check if a proxy is setup before sending the transmission, and if no proxy is setup then set it to null – Bluegreen

You'll need to examine HttpWebRequest.Proxy to see if a proxy is set, but do read the documentation because I can't remmeber the exact details: msdn.microsoft.com/en-us/library/… IIRC, the request will use the default proxy (as in the IE proxy) if defined, unless you override it to use another, or to use none. – Fanchet