Does Python csv writer always use DOS end-of-line characters?

Question

I realize that the csv library in Python always generates DOS end-of-line characters. Even if I use the 'wb' mode, even if I use Linux.

import csv

f = open('output.txt', 'wb');
writer = csv.writer(f)
writer.writerow([2,3,4]);
f.close()

The above code always uses '\r\n' as the end of line separator. How can I make it use use '\n' only?

Rockhampton

The CSV MIME spec mandates CRLF as the line delimiter: tools.ietf.org/html/rfc4180 – Ameline

I believe the csv reader always does and can't be changed, but you're only asking about the writer, whose default can be changed. – Leasia

Answer 1

You can give your writer instance a custom lineterminator argument in the constructor:

writer = csv.writer(f, lineterminator="\n")

Opalina

can lineterminator be given as combination of two characters ? – Hargis

@Hargis If you follow the link to the documentation for lineterminator, it says that it is a string, and the default value is '\r\n'. So yes. – Crossbill

Answer 2

As Niklas answered, the lineterminator argument lets you choose your line endings. Rather than hard coding it to '\n', make it platform independent by using your platform's line separator: os.linesep. Also, make sure to specify newline='' for Python 3 (see this comment).

import csv
import os

with open('output.csv', 'w', newline='') as f:
    writer = csv.writer(f, lineterminator=os.linesep)
    writer.writerow([2, 3, 4])

Old solution (Python 2 only)

In Python 2, use 'wb' mode (see the docs).

import csv
import os

with open('output.csv', 'wb') as f:
    writer = csv.writer(f, lineterminator=os.linesep)
    writer.writerow([2, 3, 4])

For others who find this post, don't miss the 'wb' if you're still using Python 2. (In Python 3, this problem is handled by Python). You won't notice a problem if you're missing it on some platforms like GNU/Linux, but it is important to open the file in binary mode on platforms where that matters, like Windows. Otherwise, the csv file can end up with line endings like \r\r\n. If you use the 'wb' and os.linesep, your line endings should be correct on all platforms.

Salmi

so even if you make the writer "platform independent", your script is still platform dependent because you need to open it in binary, but only where necessary ? If I try wb on linux I get an error for writer.writeheader() – Larimor

I just tried it on Ubuntu 16.04 using both writer and DictWriter with wb, @CiprianTomoiaga, and it works fine for me. I suggest you post a new question with a code sample and the error message, then add a comment here with a link to your question. – Salmi

@CiprianTomoiaga This doesn't work with Python 3. The CSV module assumes that it can write normal strings and this isn't the case anymore if you open the file with 'wb' (you would get a TypError). Thus, you have to open the file like this: f = open('output.csv', 'w', newline='') when you use Python 3. See also the Python 3 CSV documentation. – Grippe

Consider too that the main purpose of CSV in general is as an interchange format that works between dissimilar systems. So while using lineterminator=os.linesep will produce CSV that's suited to the system on which this script runs, that also means the CSV produced is different depending on where the script runs; neither of which is necessarily right for the system on which the CSV is consumed. So "platform independent" for the consumer might mean sticking to a single standard rather than varying with the OS of the producer. – Crossbill

While I agree that using the OS separator is normally the best option, the one place I would use "\n" is when running Python in Git Bash on Windows, or you get mixed line terminators. – Jamila

Answer 3

Python 3.2 added a unix_dialect:

The unix_dialect class defines the usual properties of a CSV file generated on UNIX systems, i.e. using '\n' as line terminator and quoting all fields. It is registered with the dialect name 'unix'.

import csv
from pathlib import Path

out_csv = Path("c:/scratch/blah.csv")
with out_csv.open("w") as f:
    csv_writer = csv.writer(f, dialect=csv.unix_dialect)
    csv_writer.writerow([1, 2, 3, 4])
    csv_writer.writerow([5, 6, 7, 8])

I've found unix_dialect imports into Google Docs with correct line endings and multiline fields. So if line endings are only one of your problems, it may solve the problems you don't yet know you have.

Puppetry