How can I Monitor HTTP Traffic and Get a List of URLs in C#?

Question

Basically, what we at the company are doing right now is plainly using HttpFox in Firefox to manually get webtrends tags and values that are appended as query string in an image/gif type url that is listed whenever we type a specific url on the browser. after starting the addon, and going to a particular URL, a list of URLs appear on the HttpFox url list. one of the URLs contains the query string we needed, and we copy that list onto the excel file afterwards. So if there are 2000 pages to get, we open those 2000 pages and do the copying and pasting one by one.

That leads to my task of automating the 'tag collection' by making a program via C# that does the sniffing/monitoring once a particular url is entered. I'll provide a list of urls to be read, and the program will do the monitoring, and fetching of data, as well as recording the data to the database.

My problem is how to fetch that list of URLs using C# once i type a specific URL - something like fiddler and httpfox do, but shouldn't be browser dependent.

Btw, Both Fiddler and HttpFox do the job that i need, but i don't think we can feed fiddler with a list of url's to sniff at. HttpFox is good but it's only run via firefox, adn importing is so manual.

HttpFox doesn't have a name for that 'list' of URLs, but under Fiddler, it's named 'Web Sessions'. Hope this one helps you understand more of my problem.

I've already looked at HttpListener and HttpWebRequest but can't seem to get the result that I am looking for.

Any help would be appreciated. :)

Marolda

Answer 1

You could embed Fiddler as a .NET library, and run it as a part of your application: http://fiddler.wikidot.com/fiddlercore

Dolomites

Hi Piskvor! Thanks! Looks like FiddlerCore is the thing that i needed. :) The problem is that I don't know how to 'setup' FiddlerCore in my VS2010 for a specific project. Thanks in advance for the help! :) – Marolda

hi piskvor! i've already made an app that uses FiddlerCore, i've recoded the console demo app and made a WinForms app using FiddlerCore. Ever used FiddlerCore before? because you could help me with my other problem. Is there a way FiddlerCore can tell when a certain webpage finished loading? - so that i can open up another webpage after that, clear the old session list, then scan the web session for the file that I needed. – Marolda

and i nearly forgot. You're a big help with the first problem. big bunch of thanks! i'm already scratching my head with that problem for more than a week already and that's my current task at my company. thanks a lot. :) – Marolda

@dev.bonski: You're welcome; you may want to post your second problem as a separate question; it's more likely that people will see it. – Dolomites

Is there any non commercial product available with a OS License? Fiddler core will have cost in a production enviroment – Ordinary