Logging raw HTTP request/response in ASP.NET MVC & IIS7

Question

I'm writing a web service (using ASP.NET MVC) and for support purposes we'd like to be able to log the requests and response in as close as possible to the raw, on-the-wire format (i.e including HTTP method, path, all headers, and the body) into a database.

What I'm not sure of is how to get hold of this data in the least 'mangled' way. I can re-constitute what I believe the request looks like by inspecting all the properties of the HttpRequest object and building a string from them (and similarly for the response) but I'd really like to get hold of the actual request/response data that's sent on the wire.

I'm happy to use any interception mechanism such as filters, modules, etc. and the solution can be specific to IIS7. However, I'd prefer to keep it in managed code only.

Any recommendations?

Edit: I note that HttpRequest has a SaveAs method which can save the request to disk but this reconstructs the request from the internal state using a load of internal helper methods that cannot be accessed publicly (quite why this doesn't allow saving to a user-provided stream I don't know). So it's starting to look like I'll have to do my best to reconstruct the request/response text from the objects... groan.

Edit 2: Please note that I said the whole request including method, path, headers etc. The current responses only look at the body streams which does not include this information.

Edit 3: Does nobody read questions around here? Five answers so far and yet not one even hints at a way to get the whole raw on-the-wire request. Yes, I know I can capture the output streams and the headers and the URL and all that stuff from the request object. I already said that in the question, see:

I can re-constitute what I believe the request looks like by inspecting all the properties of the HttpRequest object and building a string from them (and similarly for the response) but I'd really like to get hold of the actual request/response data that's sent on the wire.

If you know the complete raw data (including headers, url, http method, etc.) simply cannot be retrieved then that would be useful to know. Similarly if you know how to get it all in the raw format (yes, I still mean including headers, url, http method, etc.) without having to reconstruct it, which is what I asked, then that would be very useful. But telling me that I can reconstruct it from the HttpRequest/HttpResponse objects is not useful. I know that. I already said it.

---

Please note: Before anybody starts saying this is a bad idea, or will limit scalability, etc., we'll also be implementing throttling, sequential delivery, and anti-replay mechanisms in a distributed environment, so database logging is required anyway. I'm not looking for a discussion of whether this is a good idea, I'm looking for how it can be done.

Chloral

@Kev - No, it's a RESTful service implemented using ASP.NET MVC – Chloral

It's probably possible to do using IIS7 and a native module - msdn.microsoft.com/en-us/library/ms694280.aspx – Palladium

Have you managed to implement this? Just curious, did you adopt any buffer strategy to write to db? – Febri

Interesting project... if you do end up doing it, with final solution? – Astraphobia

Answer 1

OK, so it looks like the answer is "no you can't get the raw data, you have to reconstruct the request/response from the properties of the parsed objects". Oh well, I've done the reconstruction thing.

Chloral

Did you see Vineus' comment about ServerVariables["ALL_RAW"]? I haven't tried it myself yet, but it is documented to return the raw header information precisely as sent by the client. Even if the document turns out to be wrong, and it is doing a reconstruction, hey, free reconstruction :-) – Allyson

Answer 2

Definitely use an IHttpModule and implement the BeginRequest and EndRequest events.

All of the "raw" data is present between HttpRequest and HttpResponse, it just isn't in a single raw format. Here are the parts needed to build Fiddler-style dumps (about as close to raw HTTP as it gets):

request.HttpMethod + " " + request.RawUrl + " " + request.ServerVariables["SERVER_PROTOCOL"]
request.Headers // loop through these "key: value"
request.InputStream // make sure to reset the Position after reading or later reads may fail

For the response:

"HTTP/1.1 " + response.Status
response.Headers // loop through these "key: value"

Note that you cannot read the response stream so you have to add a filter to the Output stream and capture a copy.

In your BeginRequest, you will need to add a response filter:

HttpResponse response = HttpContext.Current.Response;
OutputFilterStream filter = new OutputFilterStream(response.Filter);
response.Filter = filter;

Store filter where you can get to it in the EndRequest handler. I suggest in HttpContext.Items. There can then get the full response data in filter.ReadStream().

Then implement OutputFilterStream using the Decorator pattern as a wrapper around a stream:

/// <summary>
/// A stream which keeps an in-memory copy as it passes the bytes through
/// </summary>
public class OutputFilterStream : Stream
{
    private readonly Stream InnerStream;
    private readonly MemoryStream CopyStream;

    public OutputFilterStream(Stream inner)
    {
        this.InnerStream = inner;
        this.CopyStream = new MemoryStream();
    }

    public string ReadStream()
    {
        lock (this.InnerStream)
        {
            if (this.CopyStream.Length <= 0L ||
                !this.CopyStream.CanRead ||
                !this.CopyStream.CanSeek)
            {
                return String.Empty;
            }

            long pos = this.CopyStream.Position;
            this.CopyStream.Position = 0L;
            try
            {
                return new StreamReader(this.CopyStream).ReadToEnd();
            }
            finally
            {
                try
                {
                    this.CopyStream.Position = pos;
                }
                catch { }
            }
        }
    }


    public override bool CanRead
    {
        get { return this.InnerStream.CanRead; }
    }

    public override bool CanSeek
    {
        get { return this.InnerStream.CanSeek; }
    }

    public override bool CanWrite
    {
        get { return this.InnerStream.CanWrite; }
    }

    public override void Flush()
    {
        this.InnerStream.Flush();
    }

    public override long Length
    {
        get { return this.InnerStream.Length; }
    }

    public override long Position
    {
        get { return this.InnerStream.Position; }
        set { this.CopyStream.Position = this.InnerStream.Position = value; }
    }

    public override int Read(byte[] buffer, int offset, int count)
    {
        return this.InnerStream.Read(buffer, offset, count);
    }

    public override long Seek(long offset, SeekOrigin origin)
    {
        this.CopyStream.Seek(offset, origin);
        return this.InnerStream.Seek(offset, origin);
    }

    public override void SetLength(long value)
    {
        this.CopyStream.SetLength(value);
        this.InnerStream.SetLength(value);
    }

    public override void Write(byte[] buffer, int offset, int count)
    {
        this.CopyStream.Write(buffer, offset, count);
        this.InnerStream.Write(buffer, offset, count);
    }
}

Shurlock

Nice answer. One comment though: you said "There can then get the full response data in filter.ToString()." -don't you mean filter.ReadStream()? (I'm implementing in vb.net not c# but if I run ToString I just get the class name as a string. .ReadStream returns the desired response body. – Pilgrim

I agree, good answer. I've used it as the basis for a custom logger but have now come across a problem where some headers are missing and most importantly when using IIS compression I can't access the final compressed response. I started a new related question (#11084959) for this. – Nymph

I think mckamey is a genius. Can you go work for Microsoft so that we just get intelligent solutions instead of needing to have brilliant workarounds? – Intolerant

Beware that request.RawUrl might trigger a request validation exception. In 4.5 you can use request.Unvalidated.RawUrl to prevent this. In 4.0 I ended up using some reflection to mimic Request.SaveAs – Mucoid

I'm running into problems similar to reading the response when reading the request -- after reading it using a StreamReader, the request.InputStream.Length is 0. After reading this reference it's just a dumb gotcha. – Probability

endurasoft.com/blog/post/… Async HttpModule has more scabality not more performance per se, isn't? – Astraphobia

@Mucoid real world samples? – Astraphobia

@Shurlock I try to implement your solution in my global.asax with Application_BeginRequest and Application_EndRequest but I'm not sure what code should I write in the EndRequest, can you provide an example in your answer plz ? – Breechcloth

I haven’t used .NET since 2010 but IIRC, the code you add to EndRequest is filter.ReadStream() where you get the result that you scraped and all of the properties. I.e, this is where you save your captured data. – Shurlock

Answer 3

The following extension method on HttpRequest will create a string that can be pasted into fiddler and replayed.

namespace System.Web
{
    using System.IO;

    /// <summary>
    /// Extension methods for HTTP Request.
    /// <remarks>
    /// See the HTTP 1.1 specification http://www.w3.org/Protocols/rfc2616/rfc2616.html
    /// for details of implementation decisions.
    /// </remarks>
    /// </summary>
    public static class HttpRequestExtensions
    {
        /// <summary>
        /// Dump the raw http request to a string. 
        /// </summary>
        /// <param name="request">The <see cref="HttpRequest"/> that should be dumped.       </param>
        /// <returns>The raw HTTP request.</returns>
        public static string ToRaw(this HttpRequest request)
        {
            StringWriter writer = new StringWriter();

            WriteStartLine(request, writer);
            WriteHeaders(request, writer);
            WriteBody(request, writer);

            return writer.ToString();
        }

        private static void WriteStartLine(HttpRequest request, StringWriter writer)
        {
            const string SPACE = " ";

            writer.Write(request.HttpMethod);
            writer.Write(SPACE + request.Url);
            writer.WriteLine(SPACE + request.ServerVariables["SERVER_PROTOCOL"]);
        }

        private static void WriteHeaders(HttpRequest request, StringWriter writer)
        {
            foreach (string key in request.Headers.AllKeys)
            {
                writer.WriteLine(string.Format("{0}: {1}", key, request.Headers[key]));
            }

            writer.WriteLine();
        }

        private static void WriteBody(HttpRequest request, StringWriter writer)
        {
            StreamReader reader = new StreamReader(request.InputStream);

            try
            {
                string body = reader.ReadToEnd();
                writer.WriteLine(body);
            }
            finally
            {
                reader.BaseStream.Position = 0;
            }
        }
    }
}

Cistaceous

Very good code! But for this to work with MVC 4 I had to change the class name to HttpRequestBaseExtensions and to change HttpRequest to HttpRequestBase at every place. – Rathenau

Answer 4

You can use the ALL_RAW server variable to get the original HTTP headers sent with the request, then you can get the InputStream as usual:

string originalHeader = HttpHandler.Request.ServerVariables["ALL_RAW"];

check out: http://msdn.microsoft.com/en-us/library/ms524602%28VS.90%29.aspx

Ulema

This worked for me too. Didn't even need to be in a handler. I was able to still access it from the page. – Fraternize

Or in the ASP.NET server context, use: this.Request.ServerVariables["ALL_RAW"]; – Liborio

I'm not able to get the request body from Request.InputStream, it returns "" for me every time, however ALL_RAW works great for returning the request headers so this answer is half right. – Roseroseann

You can also use HttpContext.Current.Request to grab the current context outside of MVC controllers, ASPX pages, etc... just make sure it's not null first ;) – Floater

Answer 5

Well, I'm working on a project and did, maybe not too deep, a log using the request params:

Take a look:

public class LogAttribute : ActionFilterAttribute
{
    private void Log(string stageName, RouteData routeData, HttpContextBase httpContext)
    {
        //Use the request and route data objects to grab your data
        string userIP = httpContext.Request.UserHostAddress;
        string userName = httpContext.User.Identity.Name;
        string reqType = httpContext.Request.RequestType;
        string reqData = GetRequestData(httpContext);
        string controller = routeData["controller"];
        string action = routeData["action"];

        //TODO:Save data somewhere
    }

    //Aux method to grab request data
    private string GetRequestData(HttpContextBase context)
    {
        StringBuilder sb = new StringBuilder();

        for (int i = 0; i < context.Request.QueryString.Count; i++)
        {
            sb.AppendFormat("Key={0}, Value={1}<br/>", context.Request.QueryString.Keys[i], context.Request.QueryString[i]);
        }

        for (int i = 0; i < context.Request.Form.Count; i++)
        {
            sb.AppendFormat("Key={0}, Value={1}<br/>", context.Request.Form.Keys[i], context.Request.Form[i]);
        }

        return sb.ToString();
    }

You can decorate your controllers class for log it entirely:

[Log]
public class TermoController : Controller {...}

or log just some individual action methods

[Log]
public ActionResult LoggedAction(){...}

Reify

Answer 6

Any reason you need to keep it in managed code?

It is worth mentioning that you can enable Failed Trace logging in IIS7 if you don't like re-inventing the wheel. This logs headers, the request and response body as well as many other things.

Ken

What if it isn't a failure? – Geilich

You can use Failed Trace logging with HTTP 200 OK as well, so non-failures can still be logged – Ken

This is by far the simplest solution. – Urbain

To see the whole stack trace, it was necessary to add GlobalConfiguration.Configuration.IncludeErrorDetailPolicy = IncludeErrorDetailPolicy.Always; at the end of Register(..) in WebApiConfig.cs, but this may vary between versions. – Dorsad

Answer 7

I went with McKAMEY's approach. Here's a module I wrote that will get you started and hopefully save you some time. You'll need to plug the Logger obviously with something that works for you:

public class CaptureTrafficModule : IHttpModule
{
    public void Init(HttpApplication context)
    {
        context.BeginRequest += new EventHandler(context_BeginRequest);
        context.EndRequest += new EventHandler(context_EndRequest);
    }

    void context_BeginRequest(object sender, EventArgs e)
    {
        HttpApplication app = sender as HttpApplication;

        OutputFilterStream filter = new OutputFilterStream(app.Response.Filter);
        app.Response.Filter = filter;

        StringBuilder request = new StringBuilder();
        request.Append(app.Request.HttpMethod + " " + app.Request.Url);
        request.Append("\n");
        foreach (string key in app.Request.Headers.Keys)
        {
            request.Append(key);
            request.Append(": ");
            request.Append(app.Request.Headers[key]);
            request.Append("\n");
        }
        request.Append("\n");

        byte[] bytes = app.Request.BinaryRead(app.Request.ContentLength);
        if (bytes.Count() > 0)
        {
            request.Append(Encoding.ASCII.GetString(bytes));
        }
        app.Request.InputStream.Position = 0;

        Logger.Debug(request.ToString());
    }

    void context_EndRequest(object sender, EventArgs e)
    {
        HttpApplication app = sender as HttpApplication;
        Logger.Debug(((OutputFilterStream)app.Response.Filter).ReadStream());
    }

    private ILogger _logger;
    public ILogger Logger
    {
        get
        {
            if (_logger == null)
                _logger = new Log4NetLogger();
            return _logger;
        }
    }

    public void Dispose()
    {
        //Does nothing
    }
}

Amanita

You cannot safely cast app.Response.Filter to anything other than a Stream. Other HttpModules may wrap your response filter with their own and in this case you will get an invalid cast exception. – Luba

Shouldn't it be Encoding.UTF8, or maybe Encoding.Default when reading the request stream? Or just use a StreamReader (with disposing caveats) – Probability

Answer 8

use a IHttpModule:

    namespace Intercepts
{
    class Interceptor : IHttpModule
    {
        private readonly InterceptorEngine engine = new InterceptorEngine();

        #region IHttpModule Members

        void IHttpModule.Dispose()
        {
        }

        void IHttpModule.Init(HttpApplication application)
        {
            application.EndRequest += new EventHandler(engine.Application_EndRequest);
        }
        #endregion
    }
}

    class InterceptorEngine
    {       
        internal void Application_EndRequest(object sender, EventArgs e)
        {
            HttpApplication application = (HttpApplication)sender;

            HttpResponse response = application.Context.Response;
            ProcessResponse(response.OutputStream);
        }

        private void ProcessResponse(Stream stream)
        {
            Log("Hello");
            StreamReader sr = new StreamReader(stream);
            string content = sr.ReadToEnd();
            Log(content);
        }

        private void Log(string line)
        {
            Debugger.Log(0, null, String.Format("{0}\n", line));
        }
    }

Dirge

Per Alex and my own experience, I don't think you can read from HttpResponse.OutputStream, so your method of logging in the ProcessResponse method probably will not work. – Guay

William is right. HttpResponse.OutputStream is not readable. I find a solution which is to use HttpResponse.Filter and replace the default output stream with your own. – Stuppy

endurasoft.com/blog/post/… async Httpmodule better ? – Astraphobia

Answer 9

if for occasional use, to get around a tight corner, how about something crude like below?

Public Function GetRawRequest() As String
    Dim str As String = ""
    Dim path As String = "C:\Temp\REQUEST_STREAM\A.txt"
    System.Web.HttpContext.Current.Request.SaveAs(path, True)
    str = System.IO.File.ReadAllText(path)
    Return str
End Function

Refraction

Answer 10

HttpRequest and HttpResponse pre MVC used to have a GetInputStream() and GetOutputStream() that could be used for that purpose. Haven't look into those part in MVC so Im not sure they are availavle but might be an idea :)

Lutanist

Answer 11

You can accomplish this in a DelegatingHandler without using the OutputFilter mentioned in other answers in .NET 4.5 using the Stream.CopyToAsync() function.

I'm not sure on the details, but it does not trigger all of the bad things that happen when you attempt to directly read the response stream.

Example:

public class LoggingHandler : DelegatingHandler
{
    protected override async Task<HttpResponseMessage> SendAsync(HttpRequestMessage request, CancellationToken cancellationToken)
    {
        DoLoggingWithRequest(request);
        var response = await base.SendAsync(request, cancellationToken);
        await DoLoggingWithResponse(response);
        return response;
    }

    private async Task DologgingWithResponse(HttpResponseMessage response) {
        var stream = new MemoryStream();
        await response.Content.CopyToAsync(stream).ConfigureAwait(false);     
        DoLoggingWithResponseContent(Encoding.UTF8.GetString(stream.ToArray()));

        // The rest of this call, the implementation of the above method, 
        // and DoLoggingWithRequest is left as an exercise for the reader.
    }
}

Compartmentalize

Answer 12

Agree with FigmentEngine, IHttpModule appears to be the way to go.

Look into httpworkerrequest, readentitybody and GetPreloadedEntityBody.

To get the httpworkerrequest you need to do this:

(HttpWorkerRequest)inApp.Context.GetType().GetProperty("WorkerRequest", bindingFlags).GetValue(inApp.Context, null);

where inApp is the httpapplication object.

Triumphant

I've already said that answer isn't suitable because it doesn't capture most of the information I asked for. How is this answer in any way helpful? – Chloral

More explain, How is this answer in any way helpful? – Astraphobia

Answer 13

I know it's not managed code, but I'm going to suggest an ISAPI filter. It's been a couple of years since I've had the "pleasure" of maintaining my own ISAPI but from what I recall you can get access to all this stuff, both before and after ASP.Net has done it's thing.

http://msdn.microsoft.com/en-us/library/ms524610.aspx

If a HTTPModule isn't good enough for what you need, then I just don't think there is any managed way of doing this in the required amount of detail. It's gonna be a pain to do though.

Nymph

Answer 14

I agree with the others, use an IHttpModule. Take a look at the answer to this question, which does almost the same thing that you are asking. It logs the request and response, but without headers.

How to trace ScriptService WebService requests?

Lyle

Answer 15

It might be best to do this outside of your application. You can set up a reverse proxy to do things like this (and much more). A reverse proxy is basically a web server that sits in your server room, and stands between your web server(s) and the client. See http://en.wikipedia.org/wiki/Reverse_proxy

Detta