Python 2.7 - Redirect handler isn't passing parameters on re-direct

Question

I have a url that I am hitting on a site that can be moved, and when the endpoint is moved, I need the POST/GET parameters to be reapplied. What am I missing to ensure this handler does this?

class RedirectHandler(urllib2.HTTPRedirectHandler):


  def http_error_301(self, req, fp, code, msg, headers):
        result = urllib2.HTTPRedirectHandler.http_error_301(
            self, req, fp, code, msg, headers)
        result.status = code
        return result

    def http_error_302(self, req, fp, code, msg, headers):
        result = urllib2.HTTPRedirectHandler.http_error_302(
            self, req, fp, code, msg, headers)
        result.status = code
        return result

When I watch the traffic through fiddler, I notice that the token, which is used for authentication gets dropped.

(please note I cannot use requests for this solution, it must be standard library only)

Thank you

Charentemaritime

Where your Cookies ? (or your session information) – Divider

@SDilmac so you think a cookie handler will solve this problem? – Charentemaritime

True ! All web services using cookie cos can't save all actions of visitor. – Divider

@SDilmac I added a cookiejar and handler to the opener, and no cookie was returned from the response server, so my guess is that the server will not return a cookie. I even tried doing the same thing (watching the traffic in fiddler) in a web browser and no authentication was returned. My guess is that the webbrowser persists the form parameters from url to url. I need to somehow on the 302 redirect capture the params/data from the original request, then push them to the new request. I tried creating a new request object, but that throws an error. – Charentemaritime

Try this code.google.com/p/python-proxy/source/browse/trunk/… . Browsers, servers, clients, ssl, and a lot more.. You want grab some header on a packet but if touch it will be corrupted. Need clone first for reuse. Or set your computer as gateway ! i hope help. – Divider

@SDilmac so you are saying when downloading, I should create a local proxy and then go through that proxy? – Charentemaritime

Yes, if you use non transparent proxy on outside. You can't grab request headers ! – Divider

Try: gist.github.com/FiloSottile/2077115 – Divider

@josh1234, obviously you've said you can't use requests, but can I ask why? Would it be possible to vendorize requests? – Clearness

Answer 1

The story about HTTP 1.0 and 1.1 status codes 302, 303 and 307 is a little complicated. Basically you see the expected and documented behaviour (you may also look at this answer for longer description):

The default implementation of this method does not strictly follow RFC 2616, which says that 301 and 302 responses to POST requests must not be automatically redirected without confirmation by the user. In reality, browsers do allow automatic redirection of these responses, changing the POST to a GET, and the default implementation reproduces this behavior.

And you go the right way, but override the wrong methods. Here's the source of urllib2.HTTPRedirectHandler.redirect_request:

def redirect_request(self, req, fp, code, msg, headers, newurl):
    """Return a Request or None in response to a redirect.
    ...
    Return None if you can't but another Handler might.
    """
    m = req.get_method()
    if (code in (301, 302, 303, 307) and m in ("GET", "HEAD")
        or code in (301, 302, 303) and m == "POST"):
        # ...
        newurl = newurl.replace(' ', '%20')
        newheaders = dict((k,v) for k,v in req.headers.items()
                          if k.lower() not in ("content-length", "content-type")
                         )
        return Request(newurl,
                       headers=newheaders,
                       origin_req_host=req.get_origin_req_host(),
                       unverifiable=True)
    else:
        raise HTTPError(req.get_full_url(), code, msg, headers, fp)

Several observations here. It doesn't pass data, thus new request is GET. It filters out content-length and content-type headers, which are required for a correct POST. If fact, in my example req.headers is an empty dict, so I resorted to req.header_items() (see unredirected_hdrs). Moreover is doesn't handle POST and 307 redirect.

Here's a correct redirector handler implementation for POST and 302 redirect. Here's also complete CherryPy simulation (do pip install cherrypy before).

#!/usr/bin/env python
# -*- coding: utf-8 -*-


import urllib2
from urllib2 import HTTPRedirectHandler, Request

import cherrypy


config = {
  'global' : {
    'server.socket_host' : '127.0.0.1',
    'server.socket_port' : 8080,
    'server.thread_pool' : 8
  }
}


class RedirectHandler(HTTPRedirectHandler):

    def redirect_request(self, req, fp, code, msg, headers, newurl):
      if code == 302 and req.get_method() == 'POST':
        return Request(newurl, headers=dict(req.header_items()), data=req.data,
          origin_req_host=req.get_origin_req_host(), unverifiable=True)
      else:
        return HTTPRedirectHandler.redirect_request(self, req, fp, code, msg, 
          headers, newurl)


class App:

  @cherrypy.expose
  def index(self):
    opener = urllib2.build_opener(RedirectHandler())
    return opener.open('http://localhost:8080/redirect', data='foo=bar')

  @cherrypy.expose
  def redirect(self, **kwargs):
    print('Before redirect {0}'.format(kwargs))
    raise cherrypy.HTTPRedirect('/target', 302)
  
  @cherrypy.expose
  def target(self, **kwargs):
    return 'Target received {0} {1}'.format(cherrypy.request.method, kwargs)


if __name__ == '__main__':
  cherrypy.quickstart(App(), '/', config)

Memorize

Thank you for the response, I will test this and get back to you! – Charentemaritime