Python 3.0 urllib.parse error "Type str doesn't support the buffer API"

Question

  File "/usr/local/lib/python3.0/cgi.py", line 477, in __init__
    self.read_urlencoded()
  File "/usr/local/lib/python3.0/cgi.py", line 577, in read_urlencoded
    self.strict_parsing):
  File "/usr/local/lib/python3.0/urllib/parse.py", line 377, in parse_qsl
    pairs = [s2 for s1 in qs.split('&') for s2 in s1.split(';')]
TypeError: Type str doesn't support the buffer API

Can anybody direct me on how to avoid this? I'm getting it through feeding data into the cgi.Fieldstorage and I can't seem to do it any other way.

Cystotomy

Answer 1

urllib is trying to do:

b'a,b'.split(',')

Which doesn't work. byte strings and unicode strings mix even less smoothly in Py3k than they used to — deliberately, to make encoding problems go wrong sooner rather than later.

So the error is rather opaquely telling you ‘you can't pass a byte string to urllib.parse’. Presumably you are doing a POST request, where the form-encoded string is coming into cgi as a content body; the content body is still a byte string/stream so it now clashes with the new urllib.

So yeah, it's a bug in cgi.py, yet another victim of 2to3 conversion that hasn't been fixed properly for the new string model. It should be converting the incoming byte stream to characters before passing them to urllib.

Did I mention Python 3.0's libraries (especially web-related ones) still being rather shonky? :-)

Tanning

Yeah. So far I've noticed huge problems with cgi, urllib, and wsgiref. I hope they get fixed soon. :( – Cystotomy

Indeed. Momentum in WEB-SIG seems to have ground to a halt; no-one seems to want ownership of the issue. Very disappointing. – Tanning

I believe this should finally be working properly in 3.2 (see bugs.python.org/issue4953). – Danner

Answer 2

From the python tutorial ( http://www.python.org/doc/3.0/tutorial/stdlib.html ) there is an example of using urlopen method. It raises the same error.

for line in urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl'):
    if 'EST' in line or 'EDT' in line:  # look for Eastern Time
        print(line)

You'll need to use the str function to convert the byte thingo to a string with the correct encoding. As follows:

for line in urlopen('http://tycho.usno.navy.mil/cgi-bin/timer.pl'):
    lineStr = str( line, encoding='utf8' )
    if 'EST' in lineStr or 'EDT' in lineStr:  # look for Eastern Time
        print(lineStr)

Profundity

It would be really interesting to provide a solution that works with both versions of python. – Naught