HTTP requests trace

Question

Are there any tools to trace the exact HTTP requests sent by a program?

I have an application which works as a client to a website and facilitates certain tasks (particularly it's a bot which makes automatic offers in a social lending webstite, based on some predefined criteria), and I'm interested in monitoring the actual HTTP requests which it makes.

Any tutorials on the topic?

Rhinestone

I'm partial to charlesproxy.com. – Reword

Answer 1

Some popular protocol/network sniffers are:

• Wireshark (previous the famous Ethereal)
• Nirsoft SmartSniff (using WinPcap)
• Nirsoft SocketSniff (allows you to watch the WinSock activity of the selected process and watch the content of each send or receive call, in Ascii mode or as Hex Dump)
• Microsoft's Network Monitor (and a list of video-tutorials here, note video 'Advanced Filtering 2 of 2' where they specifically filter on process)

Wikipedia article 'Comparison of packet analyzers' has a nice overview of some other tools to.

Alternatively you could also look into (man-in-the-middle) proxy tools like:

• Fiddler
• mitmproxy

Both of the above actually record/decrypt/modify/replay HTTPS to!! You'd need to point the application you are monitoring to this proxy. If nothing else uses that proxy the log would be application/process specific and another upside to this approach is that one could also run the monitor/logger on a different machine.

Once you choose a tool, you can easily google a tutorial to go along with it.
However the core idea is usually the same: basically one sets a filter (on capture itself or display of captured data) on things like protocol, network/mac address, portno, etc. Depending on the tool, some can also filter on local application.

Hope this helps!

Paunch

Sniffing all the entire network just for few requests coming from only 1 program, is it really the best solution? Especially if the asker wants to try it out on the production environment. – Blunk

Thanks for the info, but as far as I know, Wireshark captures everything from a chosen network interface. Is it possible to filter the requests coming from a particular program (I haven't seen such option)? – Rhinestone

@MrYoshiji: since most computers nowadays are behind a switch (that's behind a router/gateway) not a hub, one wouldn't be sniffing the 'entire network' (but in 'worst case' only all traffic that actually reaches the computer). However I understand your point and updated my answer, also addressing the askers comment. – Paunch

I tried Charles Proxy and Fiddler, and Fiddler finally did the job, as actually I needed to look into HTTPS. I've had some difficulties in configuring Charles Proxy for that purpose, but with Fiddler it went flawless. Thank you so much for your profound answer! :) – Rhinestone

Answer 2

Take a look at HTTP Toolkit (disclaimer: it's my project).

Totally automatic HTTP & HTTPS interception, with zero setup, isolated to just the code you want to debug.

You can open a browser with it, and see all the traffic from that one window immediately (but no others), or run a terminal and automatically see all traffic only from processes started from that terminal. Built-in HTTPS decryption for everything, with no risky system-wide certificates and no manual setup. Let me know what you think!

Taxicab