get a "raw" request\response from MITM Proxy

Question

i', scripting mitm proxy (http://mitmproxy.org/index.html) to write HTTP and HTTPS request and responses to a file according to their IP (each client can then access it's own requests\responses) for unit tests for mobile.

As far as i can see for now i can't just use str(Flow.request) or repr(Flow.request) to get a "raw" print of the response\request like i get in fiddler, i need to reconstruct it from the internal data of the Request and Response objects.

anyone knows of a better way ? i'm using :

def response(ScriptContext, Flow):
    Flow.request....
    Flow.response....

To access the request or response being intercepted, i'm not changing anything, just observing. For now the proxy is on 8080, later on it's to be transparent proxy on 80 and 443. If anyone has done it before i'll be happy if you can share some info.

Procrustes

Answer 1

couple of things. first youcan build the raw response yourself using str(flow.request.headers) and request.httpversion and the like. however it seems that _assemble() and _assemble_headers() do the trick just fine.

so basically:

def request(context, flow):
req = flow.request;
try:
    print("Request: -----------------");
    print(req._assemble());
    print("--------------------------");
except Exception as ee:
    print(str(ee));

def response(context, flow):
    res = flow.response;
    try:
        print("Response: -----------------");
    print(res._assemble());

    if res.content:
        size = len(res.content);
        size  = min(size, 20);
        if res.content[0:size] != res.get_decoded_content()[0:size]:
            print("\n\n");
            print(res.get_decoded_content());
    print("--------------------------");
except Exception as ee:
    print(str(ee));

as you can see if the decoded body is not similar to the non decoded one (i can check for gzip content type though) i'm printing the decoded message as well. This should be saved to files according to current dates and each file is named after the client ip taken from request\response.client_conn object. This pretty much solved my problem. Some check with fiddler shows that the request are reproducable later on which is just what i needed.

Procrustes

Thanks codeSciber! I've noticed that the output of req_assemble() does not contain fullURL of the requested resource in first line of request, it is relative path.. How can I fix that? – Shipwreck

print the host variable yourself. once in the office i can copy paste the line i use to do that, you can do that in your own script like above or change their code using your own class. – Procrustes

Hi. I am a complete noob to mitmproxy module. Need some help with #32196382. Can someone please guide me in the right direction? Also in the above answer by @Procrustes where do I call the request() from ? What is Context and flow parameters there ? – Curly

Answer 2

For those people who want to copy request/response data to clipboard while end up here:

## export the current request/response as curl/httpie/raw/request/response to clipboard
# press colon : and input one of commands and enter
export.clip curl @focus
export.clip httpie @focus
export.clip raw @focus
export.clip raw_request @focus
export.clip raw_response @focus

Mitmproxy: 5.0.1

Source code

Ensor