Change URL to another URL using mitmproxy

Question

I am trying to redirect one page to another by using mitmproxy and Python. I can run my inline script together with mitmproxy without issues, but I am stuck when it comes to changing the URL to another URL. Like if I went to google.com it would redirect to stackoverflow.com

def response(context, flow):
        print("DEBUG")
        if flow.request.url.startswith("http://google.com/"):
            print("It does contain it")
            flow.request.url = "http://stackoverflow/"

This should in theory work. I see http://google.com/ in the GUI of mitmproxy (as GET) but the print("It does contain it") never gets fired.

When I try to just put flow.request.url = "http://stackoverflow.com" right under the print("DEBUG") it won't work neither.

What am I doing wrong? I have also tried if "google.com" in flow.request.url to check if the URL contains google.com but that won't work either.

Thanks

Hulk

(1) Do you see the “DEBUG” output? (2) Is your request actually http:// or is it https://? – Kikelia

@VasiliyFaronov I see the "DEBUG" output yes (I have a separate print function, which writes to a file, which I then tail -f on, since mitmproxy's GUI fills the whole terminal). It is http:// and I have carefully copy-pasted the entire URL from the mitmproxy GUI, so it matches 100%. Obviously google.com and stackoverflow.com are just fillers. – Hulk

Alternative answer here: #24886872 – Quaternion

Answer 1

The following mitmproxy script will

1. Redirect requesto from mydomain.com to newsite.mydomain.com
2. Change the request method path (supposed to be something like /getjson? to a new one `/getxml
3. Change the destination host scheme
4. Change the destination server port

5. Overwrite the request header Host to pretend to be the origi

   import mitmproxy
   from mitmproxy.models import HTTPResponse
   from netlib.http import Headers
   def request(flow):
   
       if flow.request.pretty_host.endswith("mydomain.com"):
               mitmproxy.ctx.log( flow.request.path )
               method = flow.request.path.split('/')[3].split('?')[0]
               flow.request.host = "newsite.mydomain.com"
               flow.request.port = 8181
               flow.request.scheme = 'http'
               if method == 'getjson':
                   flow.request.path=flow.request.path.replace(method,"getxml")
               flow.request.headers["Host"] = "newsite.mydomain.com"
   

Radiolocation

do you have any url / documentation for mitmproxy how to use custom modules like this one ? – Courage

@AshfaqAhmed take a look at github.com/mitmproxy/mitmproxy/tree/master/examples – Radiolocation

Answer 2

Setting the url attribute will not help you, as it is merely constructed from underlying data. [EDIT: I was wrong, see Maximilian’s answer. The rest of my answer should still work, though.]

Depending on what exactly you want to accomplish, there are two options.

(1) You can send an actual HTTP redirection response to the client. Assuming that the client understands HTTP redirections, it will submit a new request to the URL you give it.

from mitmproxy.models import HTTPResponse
from netlib.http import Headers

def request(context, flow):
    if flow.request.host == 'google.com':
        flow.reply(HTTPResponse('HTTP/1.1', 302, 'Found',
                                Headers(Location='http://stackoverflow.com/',
                                        Content_Length='0'),
                                b''))

(2) You can silently route the same request to a different host. The client will not see this, it will assume that it’s still talking to google.com.

def request(context, flow):
    if flow.request.url == 'http://google.com/accounts/':
        flow.request.host = 'stackoverflow.com'
        flow.request.path = '/users/'

These snippets were adapted from an example found in mitmproxy’s own GitHub repo. There are many more examples there.

For some reason, I can’t seem to make these snippets work for Firefox when used with TLS (https://), but maybe you don’t need that.

Kikelia

Thanks for the reply! All I need is make the content of X look like the content of Y. So if I enter website.com on the device connected, it should show the content of another-website.com. Changing the host as written in your other example won't work, as I need the whole URL to change. google.com/account works but stackoverflow.com/account doesn't. Get what I mean? I need to replace a whole URL to another URL. – Hulk

@Snorlax Method (2) replaces the host in every requested URL. It does work for me. I’m using example.com and stackoverflow.com, and when I go to http://example.com/users in Firefox, I do actually see the Stack Overflow users page, with (relative) links pointing to other example.com pages. Maybe you can tell the actual sites you’re using? For example, if your stackoverflow.com depends on cookies, method (2) may not work, because the browser will not send stackoverflow.com cookies to google.com—but I don’t think you can do anything about that. – Kikelia

Oh, do you mean that you need to change the path in addition to the host? That’s easy, just set flow.request.path to whatever you want. – Kikelia

The problem with replacing flow.request.host is the paths need to be exactly the same. I don't want that. I want to be able to say if url == "example.com": url = "new-example.com. I know it's possible, I just don't know how. – Hulk

Still not good enough. I need to change an EXACT URL to another EXACT URL. anything-goes-here.com/anything-123.html -> google.com/account for example. – Hulk

@Snorlax An HTTP/1.1 request (to a server) does not include the full URL. It has a host and a path, separately. If you need a way to split http://google.com/account into a host and a path, then use urlparse.urlparse. – Kikelia

So basically I would check if flow.request.host.get_url() is equal to my X URL, then I would set the host and the path afterwards, which is essentially like doing X->Y? – Hulk

@Snorlax Exactly. But, as I said, depending on the site, there may be caveats like cookies, localStorage, XHR requests becoming cross-origin, etc., that would prevent the resulting page from rendering correctly. – Kikelia

This will not be an issue. All I basically need to do, is change one JSON page (file.json) into another JSON page, but it's a bit more advanced than just doing that, as I need to make it user based. – Hulk

Oh yeah I will definitely do that, but I need to reinstall my server. For some reason, mitmproxy keeps screwing up. Like I can't find mitmproxy.models. I did do sudo pip install mitmproxy, but I guess I'll have to go and compile it all from source. So far I think this sounds like a working plan! Do you know --hypasswd works? I got it to work and all that, but do you know how to read the username? Like if username == "something": set path to something else – Hulk

It works! Thanks a lot for the help! Do you know if it's possible to get the current user sending the request though? – Hulk

@Snorlax I suggest you post that as a separate question, this comment thread is too long already. – Kikelia

I agree. Thank you for the help. – Hulk

Answer 3

You can set .url attribute, which will update the underlying attributes. Looking at your code, your problem is that you change the URL in the response hook, after the request has been done. You need to change the URL in the request hook, so that the change is applied before requesting resources from the upstream server.

Antipas

I did try to set the .url "attribute" in the request hook as well, but it did not really seem to work either. – Hulk