In Perl, how to remove ^M from a file?

Question

I have a script that is appending new fields to an existing CSV, however ^M characters are appearing at the end of the old lines so the new fields end up on a new row instead of the same one. How do I remove ^M characters from a CSV file using Perl?

Fjeld

Use binmode(STDIN, ":crlf") or PERLIO=:unix:crlf (see [https://mcmap.net/q/409039/-properly-detect-line-endings-of-a-file-in-perl]). – Noblenobleman

Answer 1

You found out you can also do this:

$line=~ tr/\015//d;

Fjeld

not as readable as \r - anyone looking at that (or yourself in a year's time) would be glad of a comment stating what it does – Noli

Answer 2

^M is carriage return. You can do this:

$str =~ s/\r//g

Duaneduarchy

Answer 3

Or a 1-liner:

perl -p -i -e 's/\r\n$/\n/g' file1.txt file2.txt ... filen.txt

Woodwind

It's so easy to remember this one as Perl Pie. – Pilocarpine

On windows passing *.txt with this command does not work. It gives: Can't open *.txt: Invalid argument. Anyone? – Acidulous

No need for global 'g' as '$' matches only end of line. – Cadence

Answer 4

Slightly unrelated, but to remove ^M from the command line using Perl, do this:

perl -p -i -e "s/\r\n/\n/g" file.name

Summerlin

Answer 5

I prefer a more general solution that will work with either DOS or Unix input. Assuming the input is from STDIN:

while (defined(my $ln = <>))
  {
    chomp($ln);
    chop($ln) if ($ln =~ m/\r$/);

    # filter and write
  }

Unciform

Answer 6

This one liner replaces all the ^M characters:

dos2unix <file-name>

You can call this from inside Perl or directly on your Unix prompt.

Pamphylia

Answer 7

To convert DOS style to UNIX style line endings:

for ($line in <FILEHANDLE>) {
   $line =~ s/\r\n$/\n/;
}

Or, to remove UNIX and/or DOS style line endings:

for ($line in <FILEHANDLE>) {
   $line =~ s/\r?\n$//;
}

Shoddy

wouldn't that remove the newlines, too? – Costplus

I guess that depends on your goal. I edited to show both strategies. – Shoddy

Answer 8

This is what solved my problem. ^M is a carriage return, and it can be easily avoided in a Perl script.

while(<INPUTFILE>)
{
     chomp;
     chop($_) if ($_ =~ m/\r$/);
}

Kaffiyeh

Does that remove ^M from a CSV file? Changing the input file? Does it create some output file that will not have them? – Heisenberg

Answer 9

Little script I have for that. A modification of it helped to filter out some other non-printable characters in cross-platform legacy files.

#!/usr/bin/perl
# run this as
# convert_dos2unix.pl < input_file > output_file
undef $/;
$_ = <>;
s/\r//ge;
print;

Herold

Answer 10

perl command to convert dos line ending to unix line ending with backup of the original file:

perl -pi.bak -e 's/\r\n/\n/g' filename

This command generates filename with unix line ending and leaves the original file as filename.bak.

Vaniavanilla

Answer 11

In vi hit :.

Then s/Control-VControl-M//g.

Control-V Control-M are obviously those keys. Don't spell it out.

Featherstone

It's a bad idea to include non-printing characters like carriage return verbatim in source code like this. Far better to use the \r escape that is (a) easy to see and (b) won't get lost if the source is reformatted. – Bismuth