The sort algorithm (QuickSort) relies on a predictable IComparer implementation. After a few dozen layers of indirection in the BCL you end up at this method:
public void Sort(T[] keys, int index, int length, IComparer<T> comparer)
{
try
{
...
ArraySortHelper<T>.QuickSort(keys, index, index + (length - 1), comparer);
}
catch (IndexOutOfRangeException)
{
...
throw new ArgumentException(Environment.GetResourceString("Arg_BogusIComparer", values));
}
}
Going a bit further into the QuickSort implementation, you see code like this:
while (comparer.Compare(keys[a], y) < 0)
{
a++;
}
while (comparer.Compare(y, keys[b]) < 0)
{
b--;
}
Basically if the IComparer misbehaves the Quicksort call with throw an IndexOutOfRangeException, which is wrapped in n ArgumentException.
Here is another example of bad IComparer's
class Comparer: IComparer<int>
{
public int Compare(int x, int y)
{
return -1;
}
}
So I guess, the short answer is, anytime your IComparer implementation does not consistently compare values as defined in the documentation:
Compares two objects and returns a
value indicating whether one is less
than, equal to or greater than the
other.