Delete empty lines in csv file

Question

I have a file with 4 million of lines, every line ends with the char $, but I mistakenly add a new line after the the line delimiter while scraping a website, so right now it is looking like this:

fist name, last name, phone, address, postal code, city, region,$

$

fist name, last name, phone, address, postal code, city, region,$

$

the new line '$' only shows up of course if I use :set list, but I'm trying to use this file for a bulk insert in mysql and I'm having problems with it now.

I would like to change the file to:

fist name, last name, phone, address, postal code, city, region,$

fist name, last name, phone, address, postal code, city, region,$

How can I do this? with sed or awk or even vi ? looked up around and what I found is not really applying to this case.

please don't take in consideration the extra empty line shown above.

Thanks in advance

Duckbill

Answer 1

To remove blank lines with sed:

sed -i '/^$/d' yourfile.csv

To remove lines consisting of a single $:

sed -i '/^$$/d' yourfile.csv

Most versions of sed support the -i switch; if yours does not you will need e.g. sed '/^$$/d' yourfile.csv > newfile.csv.

Removing blank lines with white space is more complicated. This usually works:

sed '/^ *$/d' yourfile.csv

If this is not sufficient, try checking also for tabs. For older sed's, this will work:

sed '/^[ X]*$/d' yourfile.csv

where X here a tab, entered via Control-V Tab.

Newer sed's will take a [ \t\r]* or \s* or [[:space:]]*, sometimes requiring a -E switch.

Grayling

Answer 2

grep can filter lines by match (or negative match) against a regex. To exclude empty lines:

grep -v '^$' yourfile.csv > yourfile_fixed.csv

Adaptable

Thanks for the reply, I believe this should work! But I actually just found that only the first 100 lines are like this because I had to do some copy and paste with these first 100 entries. – Duckbill

Answer 3

Here are your options:

With awk:

awk 'NF' file > tmp && mv tmp file

With sed (in-place changes so make sure to backup your file using -i.bak):

sed -i '/^$/d' file

With vi:

:g/^$/d

Ambidextrous

Just loved awk 'NF' approach! – Flinders

How does the NF number-of-fields work in your example? – Immateriality

Answer 4

A blank line of a CSV file format is a comma delimiter in Linux; therefore you can delete it with the below command

sed '/^,/d' file

Penny