Python 2 and 3 csv module text-binary mode backwards compatibility

Question

I would like to create a code which is Python 2.7-3.6 compatible I am trying to fix a problem with the csv module where initially I used outfile=open('./test.csv','wb') in Python 2.7 now I have to use outfile=open('./test.csv','w') like in this question otherwise I will incur in a TypeError: a bytes-like object is required, not 'str'.

A the moment I am fixing it using this code:

import sys
w = 'w'
if sys.version_info[0] < 3:
   w = 'wb'
# Where needed
outfile=open('./test.csv',w)

Not very nice, is there any better solution for opening the file in 'wb' if I am using Python 2.7 and in w if I am using Python 3.x? To clarify I have to use wb in Python 2.7 because otherwise, I'll have a blank line every time I add a new line to a file.

Rickrickard

Answer 1

When opening files to be used with module csv on python 3 you always should add newline="" the the open statement:

import sys
mode = 'w'
if sys.version_info[0] < 3:
   mode  = 'wb'

# python 3 write 
with open("somefile.txt", mode, newline="") as f:
    pass  # do something with f

The newline parameter does not exist in python 2 - but if you skip it in python 3 you get misshaped csv output on windows with additional empty lines in it.

See csv.writer (python 3):

If csvfile is a file object, it should be opened with newline=''. If newline='' is not specified, newlines embedded inside quoted fields will not be interpreted correctly, and on platforms that use \r\n linendings on write an extra \r will be added. It should always be safe to specify newline='', since the csv module does its own (universal) newline handling.

---

You should use a contextmanaging with as well:

with open("somefile.txt", mode) as f:  # works in 2 and 3
    pass  # do something with f

to get your filehandles closed even if you run into some kind of exception. This is python 2 safe - see Methods of file objects:

It is good practice to use the with keyword when dealing with file objects. This has the advantage that the file is properly closed after its suite finishes, even if an exception is raised on the way. It is also much shorter than writing equivalent try-finally blocks.

---

Your solution - ugly but works:

import sys
python3 = sys.version_info[0] >= 3 

if python3:
    with open("somefile.txt","w",newline="") as f:
        pass
else:
    with open("somefile.txt","wb") as f:
        pass

The problem is the parameter newline does not exist in python 2. To fix that you would have to wrap/monkypath open(..) including the contextmanaging.

Allyn

Thank Patrick, yes that's why I have this problem and I want to fix it, and yes I use withstatements, I didn't put in the working example only for brevity. I think however you answer doesn't answer to my question and it's wrong because the code you wrote won't work inpython2.7. – Rickrickard

Nowhere in the question you linked is any mentioning of newline="" - there is one comment on the accepted answer that mentions it. The linked question is about 'wb' vs 'w' and str vs bytelike objects (as is yours) - not about the resulting problems of omitting newline="" like f.e. here: https://mcmap.net/q/63395/-csv-file-written-with-python-has-blank-lines-between-each-row. Going to delete this answer in a bit as it seems not related to your problems. – Allyn

I did not say anything about newline I just tested your solution and it did not solve my problem, with your edit now at least answer my question so I've upvoted, I did not want to be rude, just trying to solve a problem and give a feedback – Rickrickard

Answer 2

This is not different than what Patrick Artner suggested but might give more flexibility for argument assignment:

import sys
params = {"mode": 'w', "newline": ''} # or any other default parameters
if sys.version_info[0] < 3:
   params = {"mode": 'wb'}
# Where needed
outfile = open('./test.csv', **params)

Sigler