Removing duplicate rows in vi?

Question

I have a text file that contains a long list of entries (one on each line). Some of these are duplicates, and I would like to know if it is possible (and if so, how) to remove any duplicates. I am interested in doing this from within vi/vim, if possible.

Smoothtongued

Looks like a duplicate of #747189 – Mcneal

This one is 1 year old; that one is 10 months. So, other way around. – Smoothtongued

@Smoothtongued consensus now is to prioritize upvote count (which you also have more of): meta.stackexchange.com/questions/147643/… And those are not duplicates, that one does not mention Vim :-) – Persistence

Answer 1

If you're OK with sorting your file, you can use:

:sort u

Halfwitted

If sorting is unacceptable, use :%!uniq to simply remove duplicate entries without sorting the file. – Progressionist

once you use the command the whole file changes? how do you go back? I already saved the file by mistake ... my bad – Hermosillo

Just use Vim's undo command: u – Worldwide

@Worldwide but I already closed the file and it does not seem to remember data from the last session. (This is a downside of always closing files while saving by just pressing ZZ.) – Hermosillo

@cryptic0, uniq won't work unless the duplicates are sorted a$b$a$ does nothing – Zobe

@nilon, that's when using persistent undo comes in handy https://mcmap.net/q/144659/-using-vim-39-s-persistent-undo – Lachus

You can select the lines you want sorted and deduplicated first with V or something similar, then issue the command. – Likeness

Answer 2

Try this:

:%s/^\(.*\)\(\n\1\)\+$/\1/

It searches for any line immediately followed by one or more copies of itself, and replaces it with a single copy.

Make a copy of your file though before you try it. It's untested.

Withrow

@hop Thanks for testing it for me. I didn't have access to vim at the time. – Withrow

this hightlights all the duplicate lines for me but doesn't delete, am I missing a step here? – Transude

I'm pretty sure this will also highlight a line followed by a line that has the same "prefix" but is longer. – Uncritical

This is the better solution and doesn't change the line numbers as well. Thanks – Phosphoroscope

The only issue with this is that if you have multiple duplicates (3 or more of the same lines), you have to run this many times until all dups are gone since this only removes them one set of dups at a time. – Freeload

g/\v([^ ].*)$\n\1/d avoiding blank lines would be great – Ludeman

Another drawback of this: this won't work unless your duplicate lines are already next to each other. Sorting first would be one way of ensuring they're next to each other. At that point, the other answers are probably better. – Freeload

Answer 3

From command line just do:

sort file | uniq > file.new

Furcula

This was very handy for me for a huge file. Thanks! – Sapsago

Couldn't get the accepted answer to work, as :sort u was hanging on my large file. This worked very quickly and perfectly. Thank you! – Cercus

'uniq' is not recognized as an internal or external command, operable program or batch file. – Uncritical

Yes -- I tried this technique on a 2.3 GB file, and it was shockingly quick. – Insouciant

@Uncritical You are on windows PC? Maybe you can use cygwin. – Journal

Answer 4

awk '!x[$0]++' yourfile.txt if you want to preserve the order (i.e., sorting is not acceptable). In order to invoke it from vim, :! can be used.

Store

This is lovely! Not needing to sort is exactly what I was looking for! – Lyceum

what does it do? – Zobe

This can also be done in perl if it strikes your fancy perl -nle 'print unless $seen{$_}++' yourfile.txt – Pollinize

Answer 5

I would combine two of the answers above:

go to head of file
sort the whole file
remove duplicate entries with uniq

1G
!Gsort
1G
!Guniq

If you were interested in seeing how many duplicate lines were removed, use control-G before and after to check on the number of lines present in your buffer.

Magneto

'uniq' is not recognized as an internal or external command, operable program or batch file. – Uncritical

Answer 6

g/^\(.*\)$\n\1/d

Works for me on Windows. Lines must be sorted first though.

Banausic

This will delete a line following a line which is it's prefix: aaaa followed by aaaabb will delete aaaa erroneously. – Uncritical

Answer 7

Select the lines in visual-line mode (Shift+v), then :!uniq. That'll only catch duplicates which come one after another.

Lanam

Just to note this will only work on computers with the uniq program installed i.e. Linux, Mac, Freebsd etc – Leisurely

This will be the best answer to those who don't need sorting. And if you are windows user, consider to try Cygwin or MSYS. – Southsoutheast

Answer 8

If you don't want to sort/uniq the entire file, you can select the lines you want to make uniq in visual mode and then simply: :sort u.

Habit

If you know the line numbers you want sorted to unique you can prefix the starting and ending line numbers, eg. if you want to sort+unique lines 5 through 10 the command would be :5,10 sort u – Pollinize

Answer 9

Regarding how Uniq can be implemented in VimL, search for Uniq in a plugin I'm maintaining. You'll see various ways to implement it that were given on Vim mailing-list.

Otherwise, :sort u is indeed the way to go.

Maintop

Answer 10

I would use !}uniq, but that only works if there are no blank lines.

For every line in a file use: :1,$!uniq.

Jegar

Answer 11

:%s/^\(.*\)\(\n\1\)\+$/\1/gec

or

:%s/^\(.*\)\(\n\1\)\+$/\1/ge

this is my answer for you ,it can remove multiple duplicate lines and only keep one not remove !

Apuleius

Answer 12

This version only removes repeated lines that are contigous. I mean, only deletes consecutive repeated lines. Using the given map the function does note mess up with blank lines. But if change the REGEX to match start of line ^ it will also remove duplicated blank lines.

" function to delete duplicate lines
function! DelDuplicatedLines()
    while getline(".") == getline(line(".") - 1)
        exec 'norm! ddk'
    endwhile
    while getline(".") == getline(line(".") + 1)
        exec 'norm! dd'
    endwhile
endfunction
nnoremap <Leader>d :g/./call DelDuplicatedLines()<CR>

Ludeman

Answer 13

An alternative method that does not use vi/vim (for very large files), is from the Linux command line use sort and uniq:

sort {file-name} | uniq -u

Cavuoto

Answer 14

This command got me a buffer without any duplicate lines without sorting, and it shouldn't be very hard to research why it works or how it could work better:

:%!python3.11 -c 'exec("import fileinput\nLINES = []\nfor line in fileinput.input():\n    line = line.splitlines()[0]\n    if line not in LINES:\n        print(line)\n        LINES.append(line)\n")'

Sidewalk

Answer 15

This worked for me for both .csv and .txt

awk '!seen[$0]++' <filename> > <newFileName>

Explanation: The first part of the command prints unique rows and the second part i.e. after the middle arrow is to save the output of the first part.

awk '!seen[$0]++' <filename>

>

<newFileName>

Stela