calling uniq and sort in different orders in shell

Question

Is there a difference in the order of uniq and sort when calling them in a shell script? I’m talking here about time- and space-wise.

grep 'somePattern' | uniq | sort

vs.

grep 'somePattern' | sort | uniq

a quick test on a 140 k lines textfile showed a slight speed improvement (5.5 s vs 5.0 s) for the first method (get uniq values and then sort)

I don’t know how to measure memory usage though …

The question now is: does the order make a difference? Or is it dependent on the returned lines from grep (many/few duplicates)

Scutate

I would humbly recommend accepting a different asnwer - sort -u is the correcter way of doing this than either of your alternatives. – Cardamom

sure, but the accepted answer explains the why better – Scutate

Answer 1

The only correct order is to call uniq after sort, since the man page for uniq says:

Discard all but one of successive identical lines from INPUT (or standard input), writing to OUTPUT (or standard output).

Therefore it should be

grep 'somePattern' | sort | uniq

Ternion

I've used | uniq | sort | uniq when grepping gigabytes worth of stuff out of sorted files just to try to keep the sort from having to sort an excessive amount of data. – Portiaportico

Answer 2

I believe that sort -u is suited to this exact scenario, and will both sort and uniquify things. Obviously, that'll be more efficient than calling sort and uniq individually in either order.

Allegory

sort -u is a great hint, and no doubt, it’s more efficient than calling the two in either order. BUT, the order makes a difference (uniq | sort not working) – Scutate

In a quick test, I found that sort -u is about 7% faster than sort|uniq. – Worcestershire

Answer 3

uniq depends on the items being sorted to remove duplicates(since it compares the previous and current item), hence why sort is always run before uniq. Try it and see.

Excretion