How to remove ^M

Question

How can I remove the ^M character from a text file (at the end of line) in a Python script?

I did the following, and there are ^M at every line-break.

file = open(filename, "w")
file.write(something)

Committeewoman

Can you post more off your code?. Dont`t know where ^M comes from. New line character make and "\n" not "^M" Remeber to close "file.close()" Dont use file is an reserved word in python use my_file or something. – Lockwood

@Lockwood ^M is the terminal character escape which is written as '\n' in C syntax – Engstrom

@Ned you're right - I was thinking of the old days where if you typed Ctrl+M you get a new line, but that's because it's carriage return not newline – Engstrom

Answer 1

If you're writing the file, you should specify open(filename, "wb"). That way, you'll be writing in binary mode, and Python won't attempt to determine the correct newlines for the system you're on.

December

Thanks for your reply, but I'm trying to write 'something' to 'file' and it seems 'Universal newline mode' can only be used with 'rU', if I'm not mistaken. It doesn't work for me, anyways. – Committeewoman

Answer 2

Python can open a file in binary mode or in text mode. Text is the default, so a mode of "w" means write in text mode. In text mode, Python will adjust the line endings for the platform you're on. This means on Windows, this code:

f = open("foo.txt", "w")
f.write("Hello\n")

will result in a text file containing "Hello\r\n".

You can open the file in binary mode by using "b" in the mode:

f = open("foo.txt", "wb")
f.write("Hello\n")

results in a text file containing "Hello\n".

Blackandblue

Actually, that 'something' comes from html form textarea, where I copy and paste 'something'. The script then gets the value: <code> something = form["some_name"].value </code> – Committeewoman

Answer 3

string.replace('\r', '') worked for me.

Ugly, but nor r+ nor r+b nor NOTHING ELSE worked (for me, sure) :(

Europium

This was the only solution that worked for me. I eventually figured out why. The '\r' was not being introduced when writing the file. Instead, they were in the original file being read in. Thus this removed it, but none of the other solutions would. – Racer

Answer 4

How to explicitly set carriage return when doing json.dump? contains the solution: the open function has an extra parameter newline:

file = open(filename, "w", newline="\n")
file.write(something)

newline controls how universal newlines mode works (it only applies to text mode). It can be None, '', '\n', '\r', and '\r\n'. It works as follows:

• When reading input from the stream, if newline is None, universal newlines mode is enabled. Lines in the input can end in '\n', '\r', or '\r\n', and these are translated into '\n' before being returned to the caller. If it is '', universal newlines mode is enabled, but line endings are returned to the caller untranslated. If it has any of the other legal values, input lines are only terminated by the given string, and the line ending is returned to the caller untranslated.

• When writing output to the stream, if newline is None, any '\n' characters written are translated to the system default line separator, os.linesep. If newline is '' or '\n', no translation takes place. If newline is any of the other legal values, any '\n' characters written are translated to the given string.

Responser

Answer 5

dos2unix filename.py

to convert the line breaks to UNIX style.

Softhearted

Answer 6

For portability, you can try the following

import os
file = open(filename, "w")
file.write(something.replace('\r\n', os.linesep))

Slur

Answer 7

run autopep8 on the file

> apt-get install python-autopep8

> autopep8 python_file_name > new_python_file_name.py

Undeniable

This is not what he's asking. This would also probably have other side effects; answer the question asked with detail and real solutions. – Intercourse

Answer 8

To fix (normalize) all files in a repository you can also run

git add --renormalize .

this worked best for without having to write a script to fix the files. To make sure line endings show in git status you can set this:

git config --global core.autocrlf input

on a Mac book.

Stepup

This question is not about git. – Adlar

@Adlar That's true, but it's still a way to solve the question. I had the same problem and needed to fix multiple files, it was easier to run this git command instead of writing a python script that loops over the files and fixes them. – Stepup

I'm glad it worked for you. I don't think it's generally a good solution for the following reasons: (1) it will only work in a git repo. (2) it will not work on the output of a script unless you git add it. (3) it will not prevent CRLFs from being written, only fix them in another pass. (4) it depends on global git config (yeah sure, it needn't). (5) it does absolutely nothing that dos2unix wouldn't do only with much more effort. – Adlar