Python interface to PayPal - urllib.urlencode non-ASCII characters failing

Question

I am trying to implement PayPal IPN functionality. The basic protocol is as such:

1. The client is redirected from my site to PayPal's site to complete payment. He logs into his account, authorizes payment.
2. PayPal calls a page on my server passing in details as POST. Details include a person's name, address, and payment info etc.
3. I need to call a URL on PayPal's site internally from my processing page passing back all the params that were passed in abovem and an additional one called 'cmd' with a value of '_notify-validate'.

When I try to urllib.urlencode the params which PayPal has sent to me, I get a:

While calling send_response_to_paypal. Traceback (most recent call last):
  File "<snip>/account/paypal/views.py", line 108, in process_paypal_ipn
    verify_result = send_response_to_paypal(params)
  File "<snip>/account/paypal/views.py", line 41, in send_response_to_paypal
    params = urllib.urlencode(params)
  File "/usr/local/lib/python2.6/urllib.py", line 1261, in urlencode
    v = quote_plus(str(v))
UnicodeEncodeError: 'ascii' codec can't encode character u'\ufffd' in position 9: ordinal not in range(128)

I understand that urlencode does ASCII encoding, and in certain cases, a user's contact info can contain non-ASCII characters. This is understandable. My question is, how do I encode non-ASCII characters for POSTing to a URL using urllib2.urlopen(req) (or other method)

Details:

I read the params in PayPal's original request as follows (the GET is for testing):

def read_ipn_params(request):
    if request.POST:  
        params= request.POST.copy()  
        if "ipn_auth" in request.GET:
            params["ipn_auth"]=request.GET["ipn_auth"]
        return params
    else:  
        return request.GET.copy()  

The code I use for sending back the request to PayPal from the processing page is:

def send_response_to_paypal(params):
    params['cmd']='_notify-validate'  
    params = urllib.urlencode(params)  
    req = urllib2.Request(PAYPAL_API_WEBSITE, params)  
    req.add_header("Content-type", "application/x-www-form-urlencoded") 
    response = urllib2.urlopen(req)  
    status = response.read()  
    if not status == "VERIFIED":  
        logging.warn("PayPal cannot verify IPN responses: " + status)
        return False

    return True

Obviously, the problem only arises if someone's name or address or other field used for the PayPal payment does not fall into the ASCII range.

Cisalpine

Answer 1

Try converting the params dictionary to utf-8 first... urlencode seems to like that better than unicode:

params = urllib.urlencode(dict([k, v.encode('utf-8')] for k, v in params.items()))

Of course, this assumes your input is unicode. If your input is something other than unicode, you'll want to decode it to unicode first, then encode it:

params['foo'] = my_raw_input.decode('iso-8859-1')
params = urllib.urlencode(dict([k, v.encode('utf-8')] for k, v in params.items()))

Acuminate

You were correct - this did get rid of the Exception on URLEncode. However, now PayPal is giving me an invalid response. They are such a pain... – Cisalpine

so krys, do they (paypal) document what encoding they want, if it's not utf-8? – Quotation

Thanks! Minor bug: there's a ] too much at the end of the first exampe you give. – Dugger

Thanks for this great answer. It really helped me solve the problem that I was having with PayPal IPN -- however in my case there were two more issues that tripped me up for a while. (1) it is necessary to tell PayPal to transmit using UTF-8. This can be done by entering in PayPal->Profile->My Selling Tools->PayPal button language encoding->More Options->UTF-8. (2) Additionally, it is worth pointing out that in the Sandbox, selecting the UTF-8 coding did not have any effect (it continued to use some other encoding), but this worked in the production environment. – Ivanovo

Thanks much for posting your solution, this also works with django satchmo for anyone who might be having problems with that. – Selfrestraint

Answer 2

Instead of encoding to utf-8, one should encode to what ever the paypal is using for the post. It is available under key 'charset' in the form paypal sends.

So the following code worked for me:

data = dict([(k, v.encode(data['charset'])) for k, v in data.items()])

Kimikokimitri

Answer 3

I know it's a bit late to chime in here, but the best solution I found was to not even parse what they were giving back. In django (don't know what you're using) I was able to get the raw request they sent, which I passed back verbatim. Then it was just a matter of putting the cmd key onto that.

This way it never matters what encoding they send you, you're just sending it right back.

Superload

Have you tried this with an IPN response that includes unicode characters such as ñ, á, é, etc.? The problem only occurs with these types of characters. – Ivanovo