MITMPROXY make output human readable to file

Question

My system:

Ubuntu 16.04
Mitmproxy version 3.0.4
Python version 3.5.2

I've successfully installed mitmproxy from: docs.mitmproxy.org on my server. But now I got confused how to save log mitmproxy to file? I try use mitmdump --mode transparent --showhost -p 9001 -w output_file

While I open output_file, it's not human readable. I read the docs and try scripts from the mitmproxy's Github, but no clue.

Anyone know how to save log mitmproxy to file, but human readable?
Thank you!

Beech

Answer 1

As you have probably noticed, mitmproxy generates streams in a binary format. If you want to save the streams in a human readable format, you can pass a script when you run mitmproxy to do so.

save.py

from mitmproxy.net.http.http1.assemble import assemble_request, assemble_response

f = open('/tmp/test/output.txt', 'w')

def response(flow):
    f.write(assemble_request(flow.request).decode('utf-8'))

And now run mitmproxy -s save.py and the output will be written to output.txt in a human readable format.

Do pay attention to the responses because they might contain a lot of binary data. but if you do want to write the responses also in a human readable format, then you can add f.write(assemble_response(flow.response).decode('utf-8', 'replace')) to the script.

Example output from the script:

❯❯ tail -f output.txt
GET / HTTP/1.1
Host: example.com
User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:77.0) Gecko/20100101 Firefox/77.0
Accept: text/html,application/xhtml+xml,application/xml;q=0.9,image/webp,*/*;q=0.8
Accept-Language: en-US,en;q=0.5
Accept-Encoding: gzip, deflate
Connection: keep-alive
Upgrade-Insecure-Requests: 1
If-Modified-Since: Thu, 17 Oct 2019 07:18:26 GMT
If-None-Match: "3147526947"
Cache-Control: max-age=0

Bickering

how would I transform an existing file? – Boote

Is it possible to reverse this as well? As in, convert the binary to human readable format and then revert it back to binary ? – Dogs

Answer 2

I initially had trouble getting the requests and responses output to file using what @securisec proposed but what worked for me was to replace mitmproxy -s save.py with mitmdump -s save.py instead.

Once I killed the mitmdump process, the output file was written to the file system.

Pede