Detecting if internet connection is busy

Question

We are developing an application that will install on PC and it will perform some background upload and download to/from our server. One of the requirement is to detect if the internet connection is currently busy (say above 50% utilization) and if it is, it needs to back-off and try another time. The main reason is to ensure the app does not interfere with user experience if they are in the middle of gaming, watching online movie or aggressively downloading files

After much thinking and research on google and of course SO, I still haven't found a good way on how to implement this, so decided to throw this out here. The application is implemented in C#, .NET 4.0 and I am looking for all forms of responses - either implementation in C# or other languages, pseudo-logic or approach on how to achieve - measuring of internet traffic utilization on local PC with good enough accuracy.

To avoid duplication of effort, so far I have tried these (and why they aren't suitable)

• Use WMI to get network statistic. Most SO posts and solutions out there since to refer to this as the approach but it doesn't meet our requirement as measuring of bytes sent/received against network interface capacity (e.g. 1GB Ethernet Card) for utilisation will yield a good measure for LAN traffic but not for internet traffic (where the actual internet bandwidth might only be say 8Mbps)
• Use of .NET Network Information Statistics or performance counter - yield similar readings to the above hence have the same shortcomings
• Use ICMP (Ping) and measure RTT. It was suggested that 400ms RTT is considered as slow and good indication for busy network, however I was told that user with modem (yes we have to support that), use of reverse proxy or microwave link often get ping above that hence not a good measure
• Start downloading a known file and measure the speed - this itself generate traffic which we are trying to avoid, also if this check is done often enough, our application will end up creating a lot of internet traffic - which again not ideal
• MOD: Using BITS - this service can be disabled on user pc, requires group policy changes and assume server to be IIS (with custom configuration) and in our case our server is not IIS

So here it is, I'm all confuse and looking for some advice. I highlighted the question text so that you guys don't get lost reading this and wondering what the question is. Thanks.

Froward

The correct solution would be QoS on the router. Pity that most consumer routers suck. – Calorie

Seem like a common problem, but seem like no common solution :( – Froward

Because the problem is impossible to solve at the PC level. Your software can't know how much traffic somebody else, using the same router, is producing. – Calorie

I'm very much in agreement with you. I thought before I give up, I should see if others think the same way or may have solved this problem in the past with some clever idea and willing to share their experience. – Froward

Wouldn't a more sensible approach be to have a base reference like a timeout value that if your uploading or downloading starts to take longer than N seconds you throttle it and then try again at full speed after N number of seconds? – Tumular

@Llyod - the thought was to avoid the resource contention back-off before if possible rather than tried-failed-backoff – Froward

You could end up artificially throttling your access when you don't need to - imagine if a number of programs were all trying to be nice - their combined traffic adds up to, say, 51%. So one of them drops out - then spots that the traffic is <50%, so starts up again. Either the same program or another one then spots the issue, and drops out... – Edaphic

Answer 1

You could use UPnP to query the router, and retrive the number of bytes sent and received over the network. You could keep checking this value on the router to determine what the activity is. Unfortunately this functionality doesn't seem to be well-documented, but it is possible to implement UPnP communication functionality within a C# application. You will need to use UDP to query for the router (UPnP discover), and once you have found the device, query its functionality, and then query the number of packets sent and received for the Internet Gateway Device using a WebClient (TCP).

Code for a UPnP library:

using System;
using System.Collections.Generic;
using System.Text;
using System.Net.Sockets;
using System.Net;
using System.Xml;
using System.IO;
namespace UPNPLib
{
    public class RouterElement
    {
        public RouterElement()
        {
        }
        public override string ToString()
        {
            return Name;
        }
        public List children = new List();
        public RouterElement parent;
        public string Name;
        public string Value;
        public RouterElement this[string name] {
            get
            {
                foreach (RouterElement et in children)
                {
                    if (et.Name.ToLower().Contains(name.ToLower()))
                    {
                        return et;
                    }
                }
                foreach (RouterElement et in children)
                {
                    Console.WriteLine(et.Name);
                }
                throw new KeyNotFoundException("Unable to find the specified entry");
            }
    }
        public RouterElement(XmlNode node, RouterElement _parent)
        {

            Name = node.Name;
            if (node.ChildNodes.Count 
        /// Gets the root URL of the device
        /// 
        /// 
        public static string GetRootUrl()
        {
            StringBuilder mbuilder = new StringBuilder();
            mbuilder.Append("M-SEARCH * HTTP/1.1\r\n");
            mbuilder.Append("HOST: 239.255.255.250:1900\r\n");
            mbuilder.Append("ST:upnp:rootdevice\r\n");
            mbuilder.Append("MAN:\"ssdp:discover\"\r\n");
            mbuilder.Append("MX:3\r\n\r\n");
            UdpClient mclient = new UdpClient();
            byte[] dgram = Encoding.ASCII.GetBytes(mbuilder.ToString());
            mclient.Send(dgram,dgram.Length,new IPEndPoint(IPAddress.Broadcast,1900));
            IPEndPoint mpoint = new IPEndPoint(IPAddress.Any, 0);
            rootsearch:

            dgram = mclient.Receive(ref mpoint);
            string mret = Encoding.ASCII.GetString(dgram);
            string orig = mret;
            mret = mret.ToLower();
            string url = orig.Substring(mret.IndexOf("location:") + "location:".Length, mret.IndexOf("\r", mret.IndexOf("location:")) - (mret.IndexOf("location:") + "location:".Length));
            WebClient wclient = new WebClient();
            try
            {
                Console.WriteLine("POLL:" + url);
                string reply = wclient.DownloadString(url);

                if (!reply.ToLower().Contains("router"))
                {
                    goto rootsearch;
                }
            }
            catch (Exception)
            {
                goto rootsearch;
            }
            return url;
        }
        public static RouterElement enumRouterFunctions(string url)
        {

            XmlReader mreader = XmlReader.Create(url);
            XmlDocument md = new XmlDocument();
            md.Load(mreader);
            XmlNodeList rootnodes = md.GetElementsByTagName("serviceList");
            RouterElement elem = new RouterElement();
            foreach (XmlNode et in rootnodes)
            {
                RouterElement el = new RouterElement(et, null);
                elem.children.Add(el);
            }

            return elem;
        }
        public static RouterElement getRouterInformation(string url)
        {
            XmlReader mreader = XmlReader.Create(url);
            XmlDocument md = new XmlDocument();
            md.Load(mreader);
            XmlNodeList rootnodes = md.GetElementsByTagName("device");
            return new RouterElement(rootnodes[0], null);
        }

    }
    public class RouterMethod
    {
        string url;
        public string MethodName;
        string parentname;
        string MakeRequest(string URL, byte[] data, string[] headers)
        {
            Uri mri = new Uri(URL);
            TcpClient mclient = new TcpClient();
            mclient.Connect(mri.Host, mri.Port);
            Stream mstream = mclient.GetStream();
            StreamWriter textwriter = new StreamWriter(mstream);
            textwriter.Write("POST "+mri.PathAndQuery+" HTTP/1.1\r\n");

            textwriter.Write("Connection: Close\r\n");

            textwriter.Write("Content-Type: text/xml; charset=\"utf-8\"\r\n");

            foreach (string et in headers)
            {
                textwriter.Write(et + "\r\n");
            }
            textwriter.Write("Content-Length: " + (data.Length).ToString()+"\r\n");
            textwriter.Write("Host: " + mri.Host+":"+mri.Port+"\r\n");


            textwriter.Write("\r\n");
            textwriter.Flush();



            Stream reqstream = mstream;
            reqstream.Write(data, 0, data.Length);
            reqstream.Flush();
            StreamReader reader = new StreamReader(mstream);
            while (reader.ReadLine().Length > 2)
            {

            }
            return reader.ReadToEnd();
        }
        public RouterElement Invoke(string[] args)
        {

            MemoryStream mstream = new MemoryStream();
            StreamWriter mwriter = new StreamWriter(mstream);
            //TODO: Implement argument list
            string arglist = "";

            mwriter.Write("" + "" + "");


            mwriter.Write("");//" + arglist + "");
            mwriter.Write("");
            mwriter.Flush();

            List headers = new List();

            headers.Add("SOAPAction: \"" + parentschema + "#" + MethodName + "\"");

            mstream.Position = 0;
            byte[] dgram = new byte[mstream.Length];

            mstream.Read(dgram, 0, dgram.Length);

            XmlDocument mdoc = new XmlDocument();
            string txt = MakeRequest(url, dgram, headers.ToArray());
            mdoc.LoadXml(txt);
            try
            {
                RouterElement elem = new RouterElement(mdoc.ChildNodes[0], null);

                return elem["Body"].children[0];
            }
            catch (Exception er)
            {
                RouterElement elem = new RouterElement(mdoc.ChildNodes[1], null);
                return elem["Body"].children[0];
            }

        }
        public List parameters = new List();
        string baseurl;
        string parentschema;
        public RouterMethod(string svcurl, RouterElement element,string pname, string baseURL, string svcpdsc)
        {
            parentschema = svcpdsc;
            baseurl = baseURL;
            parentname = pname;
            url = svcurl;
            MethodName = element["name"].Value;
            try
            {
                foreach (RouterElement et in element["argumentList"].children)
                {
                    parameters.Add(et.children[0].Value);
                }
            }
            catch (KeyNotFoundException)
            {
            }
        }
    }
    public class RouterService
    {
        string url;
        public string ServiceName;
        public List methods = new List();
        public RouterMethod GetMethodByNonCaseSensitiveName(string name)
        {
            foreach (RouterMethod et in methods)
            {
                if (et.MethodName.ToLower() == name.ToLower())
                {
                    return et;
                }
            }
            throw new KeyNotFoundException();
        }
        public RouterService(RouterElement element, string baseurl)
        {

            ServiceName = element["serviceId"].Value;
            url = element["controlURL"].Value;

            WebClient mclient = new WebClient();
            string turtle = element["SCPDURL"].Value;
            if (!turtle.ToLower().Contains("http"))
            {
                turtle = baseurl + turtle;
            }
            Console.WriteLine("service URL " + turtle);
            string axml = mclient.DownloadString(turtle);
            XmlDocument mdoc = new XmlDocument();
            if (!url.ToLower().Contains("http"))
            {
                url = baseurl + url;
            }
            mdoc.LoadXml(axml);
            XmlNode mainnode = mdoc.GetElementsByTagName("actionList")[0];
            RouterElement actions = new RouterElement(mainnode, null);
            foreach (RouterElement et in actions.children)
            {
                RouterMethod method = new RouterMethod(url, et,ServiceName,baseurl,element["serviceType"].Value);
                methods.Add(method);
            }

        }
    }
}

Code for a bandwidth meter:
using System;
using System.Collections.Generic;
using System.ComponentModel;
using System.Data;
using System.Drawing;
using System.Text;
using System.Windows.Forms;
using UPNPLib;
using System.IO;

namespace bandwidthmeter
{
    public partial class Form1 : Form
    {
        public Form1()
        {
            InitializeComponent();
            BinaryReader mreader = new BinaryReader(File.Open("bandwidthlog.txt", FileMode.OpenOrCreate));
            if (mreader.BaseStream.Length > 0)
            {
                prevsent = mreader.ReadInt64();
                prevrecv = mreader.ReadInt64();
            }
            mreader.Close();
            List services = new List();
            string fullurl = UPNP.GetRootUrl();
            RouterElement router = UPNP.enumRouterFunctions(fullurl);
            Console.WriteLine("Router feature enumeration complete");
            foreach (RouterElement et in router.children)
            {

                services.Add(new RouterService(et.children[0], fullurl.Substring(0, fullurl.IndexOf("/", "http://".Length+1))));
            }
            getReceiveDelegate = services[1].GetMethodByNonCaseSensitiveName("GetTotalBytesReceived");
            getSentDelegate = services[1].GetMethodByNonCaseSensitiveName("GetTotalBytesSent");
            Console.WriteLine("Invoking " + getReceiveDelegate.MethodName);
            //Console.WriteLine(services[1].GetMethodByNonCaseSensitiveName("GetTotalPacketsSent").Invoke(null));

            Timer mymer = new Timer();
            mymer.Tick += new EventHandler(mymer_Tick);
            mymer.Interval = 1000;
            mymer.Start();
            FormClosed += new FormClosedEventHandler(Form1_FormClosed);
        }
        long prevsent = 0;
        long prevrecv = 0;
        void Form1_FormClosed(object sender, FormClosedEventArgs e)
        {
            BinaryWriter mwriter = new BinaryWriter(File.Open("bandwidthlog.txt", FileMode.OpenOrCreate));
            mwriter.Write(getsent());
            mwriter.Write(getreceived());
            mwriter.Flush();
            mwriter.Close();

        }
        long getsent()
        {
            long retval = Convert.ToInt64(getSentDelegate.Invoke(null).children[0].Value);
            if (prevsent > retval)
            {
                retval = prevsent + retval;
            }
            return retval;
        }
        long getreceived()
        {
            long retval = Convert.ToInt64(getReceiveDelegate.Invoke(null).children[0].Value);
            if (prevrecv > retval)
            {
                retval = prevrecv + retval;
            }
            return retval;
        }
        void mymer_Tick(object sender, EventArgs e)
        {
            label1.Text = "Sent: "+(getsent()/1024/1024).ToString()+"MB\nReceived: "+(getreceived()/1024/1024).ToString()+"MB";

        }
        RouterMethod getSentDelegate;
        RouterMethod getReceiveDelegate;

    }
}

Wallet

Thanks for the response. Will this approach work with all supported Win OS and most routers? Also once I get the packet sent and received, how to do I about determining the utilization (like in my example > 50% is consider busy) – Froward

Yes, this approach will work with most consumer routers, but not most business ones (UPnP is disabled by default on corporate routers). – Wallet

Please read this article - codeproject.com/Articles/27992/NAT-Traversal-with-UPnP-in-C for an introduction to UPnP. This sample is intended for port forwarding, but you can use the URLs that the router provides to look up its other functions (the bandwidth functions are usually the same across most routers). – Wallet

+1 Thanks for the sample and info provided. Once we have the byte sent/received as in your code sample, I am still unclear on how do you determine the utilization percentage? – Froward

@FadrianSudaman You would need to have the user select their amount of bandwidth, similar to the way Free Download Manager does it (they ask the user how much bandwidth is available), or you could use a 3rd party site API to run a speed test. – Wallet

After some analysis, the posted answer hasn't really solved the problem as this will not cater for multiple users sharing router, I still can't derive the bandwidth capacity without simulating speedtest and there is a risk that devices that our users have doesnt support this. Saying all that, I really appreciate your time and sharing this. I learnt something from this answer and will accept this. Many thanks. – Froward

Answer 2

Have you considered using Background Intelligent Transfer Service (BITS). It's designed to do this job already:

Background Intelligent Transfer Service (BITS) transfers files (downloads or uploads) between a client and server and provides progress information related to the transfers. You can also download files from a peer.

and,

Preserve the responsiveness of other network applications.

I'm not sure if there's a managed interface to it (I can see reference to Powershell Cmdlets), so you might have to use COM interop to use it.

Edaphic

Yes I have but unfortunately it doesn't meet our need due to multiple reasons: BITS service can be disabled, it requires group policy changes and the server end point (assumed IIS) and need to be configured - in our case the server isn't IIS. +1 for suggestion. – Froward

Answer 3

Making the assumption that you are targetting Windows PC's (as you said you were developing in C#), have you looked at BITS, the Background Intelligent Transfer Service?

There's examples of how to hook into it using C# on MSDN and elsewhere, e.g. http://msdn.microsoft.com/en-us/magazine/cc188766.aspx

Earthy

Thanks for the feedback. Yes I have, please see my response to Damien answer which touched on the same thing. – Froward