Sort & uniq in Linux shell

Question

What is the difference between the following two commands?

sort -u FILE

sort FILE | uniq

Silly

When you ran them, what did you see? Did you try collecting timing differences for different sized files? You could run a few experiments and post the results are part of your question. – Bonnette

I want to know if there is a special case were both commands behave differently, in a normal execution they both give the same results – Silly

What is the difference between “sort -u” and “sort | uniq”? – Norahnorbert

@mtk: The Q&A on U&L is essentially a duplicate of this, but was asked years after this question. The historical commentary in the answer is interesting. The -u option was in UNIX 7th Edition sort (circa 1979), so the ancient history referred to is truly ancient. – Absquatulate

@Jonathan Well I posted it for the same reason that the historical commentary is interesting :) Plus it has some timing experiment illustrated. – Norahnorbert

Answer 1

Using sort -u does less I/O than sort | uniq, but the end result is the same. In particular, if the file is big enough that sort has to create intermediate files, there's a decent chance that sort -u will use slightly fewer or slightly smaller intermediate files as it could eliminate duplicates as it is sorting each set. If the data is highly duplicative, this could be beneficial; if there are few duplicates in fact, it won't make much difference (definitely a second order performance effect, compared to the first order effect of the pipe).

Note that there times when the piping is appropriate. For example:

sort FILE | uniq -c | sort -n

This sorts the file into order of the number of occurrences of each line in the file, with the most repeated lines appearing last. (It wouldn't surprise me to find that this combination, which is idiomatic for Unix or POSIX, can be squished into one complex 'sort' command with GNU sort.)

There are times when not using the pipe is important. For example:

sort -u -o FILE FILE

This sorts the file 'in situ'; that is, the output file is specified by -o FILE, and this operation is guaranteed safe (the file is read before being overwritten for output).

Absquatulate

Thanks for the complete answer! – Silly

gnu sort does not have a way to do all of sort | uniq -c | sort -n, and neither have I found any other tool to do it efficiently. Seems like a worthwhile thing to code up. – Janeljanela

Answer 2

There is one slight difference: return code.

The thing is that unless shopt -o pipefail is set the return code of the piped command will be return code of the last one. And uniq always returns zero (success). Try examining exit code, and you'll see something like this (pipefail is not set here):

pavel@lonely ~ $ sort -u file_that_doesnt_exist ; echo $?
sort: open failed: file_that_doesnt_exist: No such file or directory
2
pavel@lonely ~ $ sort file_that_doesnt_exist | uniq ; echo $?
sort: open failed: file_that_doesnt_exist: No such file or directory
0

Other than this, the commands are equivalent.

Suggest

Answer 3

Beware! While it's true that "sort -u" and "sort|uniq" are equivalent, any additional options to sort can break the equivalence. Here's an example from the coreutils manual:

For example, 'sort -n -u' inspects only the value of the initial numeric string when checking for uniqueness, whereas 'sort -n | uniq' inspects the entire line.

Similarly, if you sort on key fields, the uniqueness test used by sort won't necessarily look at the entire line anymore. After being bitten by that bug in the past, these days I tend to use "sort|uniq" when writing Bash scripts. I'd rather have higher I/O overhead than run the risk that someone else in the shop won't know about that particular pitfall when they modify my code to add additional sort parameters.

Brookebrooker

Answer 4

sort -u will be slightly faster, because it does not need to pipe the output between two commands

also see my question on the topic: calling uniq and sort in different orders in shell

Enneahedron

Answer 5

I have worked on some servers where sort don't support '-u' option. there we have to use

sort xyz | uniq

Coact

Would you care to specify which servers with which o/s version and when, roughly? The 7th Edition UNIX™ sort supported -u and that was the first widely used version of UNIX, so all others (System III, System V, BSD, etc) tended to follow it, so I'd be surprised indeed to find a Unix-like system where sort did not support -u. – Absquatulate

Answer 6

Nothing, they will produce the same result

Shiftless