TypeError: a bytes-like object is required, not 'str' in python and CSV

Question

TypeError: a bytes-like object is required, not 'str'

I'm getting the above error while executing the below python code to save the HTML table data in a CSV file. How do I get rid of that error?

import csv
import requests
from bs4 import BeautifulSoup

url='http://www.mapsofindia.com/districts-india/'
response=requests.get(url)
html=response.content

soup=BeautifulSoup(html,'html.parser')
table=soup.find('table', attrs={'class':'tableizer-table'})
list_of_rows=[]
for row in table.findAll('tr')[1:]:
    list_of_cells=[]
    for cell in row.findAll('td'):
        list_of_cells.append(cell.text)
    list_of_rows.append(list_of_cells)
outfile=open('./immates.csv','wb')
writer=csv.writer(outfile)
writer.writerow(["SNo", "States", "Dist", "Population"])
writer.writerows(list_of_rows)

Motorist

Possible duplicate of python 3.5: TypeError: a bytes-like object is required, not 'str' – Metatarsal

Answer 1

You are using Python 2 methodology instead of Python 3.

Change:

outfile=open('./immates.csv','wb')

To:

outfile=open('./immates.csv','w')

and you will get a file with the following output:

SNo,States,Dist,Population
1,Andhra Pradesh,13,49378776
2,Arunachal Pradesh,16,1382611
3,Assam,27,31169272
4,Bihar,38,103804637
5,Chhattisgarh,19,25540196
6,Goa,2,1457723
7,Gujarat,26,60383628
.....

In Python 3 csv takes the input in text mode, whereas in Python 2 it took it in binary mode.

Edited to Add

Here is the code I ran:

url='http://www.mapsofindia.com/districts-india/'
html = urllib.request.urlopen(url).read()
soup = BeautifulSoup(html)
table=soup.find('table', attrs={'class':'tableizer-table'})
list_of_rows=[]
for row in table.findAll('tr')[1:]:
    list_of_cells=[]
    for cell in row.findAll('td'):
        list_of_cells.append(cell.text)
    list_of_rows.append(list_of_cells)
outfile = open('./immates.csv','w')
writer=csv.writer(outfile)
writer.writerow(['SNo', 'States', 'Dist', 'Population'])
writer.writerows(list_of_rows)

Lemuel

For use with the csv module, the Python 3 open should also have newline='' as a parameter [ref] – Theotokos

Change the 'wb' string to 'w' works for me. Thanks a lot – Upthrow

If you are using a buffer see vinyll's answer! – Muskellunge

hi there - i tried out the code - and got back this:` Traceback (most recent call last): File "/home/martin/.atom/python/examples/bs_gumtree_pl.py", line 20, in <module> UnicodeEncodeError: 'ascii' codec can't encode character u'\xa0' in position 0: ordinal not in range(128) [Finished in 1.415s]` i have no glue what goes on here – Cowboy

Is there a python2 and python3 compatible way to do this? (I think that whenever a question addresses an issue that results from differences between python2 and python3 it is nice to give a robust version that works in both cases.) – Bonita

Answer 2

I had the same issue with Python3. My code was writing into io.BytesIO().

Replacing with io.StringIO() solved.

Calabash

happens to me with stringio also – Nonfiction

One consideration: io.StringIO() is memory greed and can be a headache with large files. – Umberto

Answer 3

just change wb to w

outfile=open('./immates.csv','wb')

to

outfile=open('./immates.csv','w')

Methylnaphthalene

Answer 4

You are opening the csv file in binary mode, it should be 'w'

import csv

# open csv file in write mode with utf-8 encoding
with open('output.csv','w',encoding='utf-8',newline='')as w:
    fieldnames = ["SNo", "States", "Dist", "Population"]
    writer = csv.DictWriter(w, fieldnames=fieldnames)
    # write list of dicts
    writer.writerows(list_of_dicts) #writerow(dict) if write one row at time

Repetitious

Answer 5

file = open('parsed_data.txt', 'w')
for link in soup.findAll('a', attrs={'href': re.compile("^http")}): print (link)
soup_link = str(link)
print (soup_link)
file.write(soup_link)
file.flush()
file.close()

In my case, I used BeautifulSoup to write a .txt with Python 3.x. It had the same issue. Just as @tsduteba said, change the 'wb' in the first line to 'w'.

Geocentric

When giving an answer it is preferable to give some explanation as to WHY your answer is the one. In this case how does this answer differ from the accepted answer? – Fulminate

@StephenRauch Thank you for your comments. I'm new here and just have started learning Python several weeks ago. I will try to give better answer in the future. – Geocentric

You can edit this post, and add more detail. Hit the edit button below and to the left of the post. – Fulminate

@StephenRauch Thanks for your tips! – Geocentric