IE versions prior to IE10 has just a single Quirks Mode. This mode is now known as IE5 Quirks In IE10 and higher.
IE10 introduced a second Quirks mode which it simply calls Quirks.
Both of these modes change the rendering mode to use the old "quirks" box model, among other layout changes. This much is consistent between them.
The original Quirks Mode was basically a backward-compatibility mode to emulate IE5. Because of this, in addition to changing the box model, etc, it also disabls the majority of the browser features that have been introduced since IE5 -- so in this mode, none of the new HTML5 features will work.
The newer Quirks mode uses the same rendering rules as the old quirks mode, but does not disable all the browser features.
That is the basic difference between them.
The question could then be asked as to why bother having both modes?
The answer to this is that other browsers (eg Chrome) also have a Quirks mode, which is triggered (just like IE) when a page doesn't have a doctype. But other browsers have never disabled any of their features for backward compatibility in the way that IE did.
The result was that a site in quirks mode could look quite different in IE compared with other browsers, even when the basic rendering rules were the same.
The newer quirks mode was therefore introduced by IE in an effort to improve cross-browser compatibility for sites in quirks mode.
But as a result, the newer quirks mode will render some pages differently to the older quirks mode, so the older one was still needed in order for MS to support users whose sites were written for it.
Ultimately one should, if at all possible, avoid using quirks mode (either of them). Despite the cross-browser efforts MS put in by introducing the extra mode, there are still major differences between browsers when you are in quirks mode.
Best practice is always to use Standards mode, wherever possible.
Quirks mode should be unnecessary anyway: If you need the Quirks mode box model, this can be accessed in standards mode by using CSS box-sizing:border-box
.