As it was already said, the algorithm is called introsort, a combination between Insertionsort, Quicksort & Heapsort.
The Problem with quicksort is, that It has a very bad worst-case complexity of O(n²), but needs only O(log(n)) extra storage compared to mergesort with O(n).
As pseudocode, it would look like this:
input : unsorted list
output : sorted list
def void sort( L ):
depthLimit = 2*floor(log(L))
introsort(L, depthLimit);
def void introsort(L , depthLimit):
n = |L|;
if n ≤ 16 then
insertionSort(L);
else if depthLimit == 0 then
heapsort(L);
else
p = quicksort_partition(A);
introsort(L[0:p-1], depthLimit - 1);
introsort(L[p+1:n], depthLimit - 1);
The main idea here is to use quicksort when it is not reaching its worst case complexity, if It does, using heapsort instead until we got part lists of the length 16, and use Insertionsort after then.
The worst and avg complexity is O(n*log(n)) which is as good as mergesort, but does not come with the problem of the linear extra storage.
The sorting algorithm is not stable, that is the only compromise you have to make compared to mergesort-