In HTML (including XHTML) as used on web pages, DOCTYPE is a string that triggers one of a few browser modes (quirks mode, standards mode, almost standards mode), depending on the exact spelling of the DOCTYPE. You want to use it to select a browser mode that best suits your page.
Formally, in SGML and XML, a DOCTYPE declaration is a reference to a Document Type Definition (DTD), which specifies the formal syntax rules of the markup language. No browser has ever used DTDs for anything or even accessed them. However, they are used by SGML and XML markup validators such as the W3C Markup Validator, except in HTML5 mode. Therefore, the choice of DOCTYPE determines how a validator works if the document is submitted to it. However, the validator mode of operation can also be selected in its user interface. (SGML and XML processors may use DOCTYPEs in different other ways, too, but the question is apparently meant to be limited to the HTML context and to web browsers and closely related software.)
There is no authoritative list of DOCTYPEs. Each HTML specification or draft defines its own DOCTYPE, or DOCTYPEs. The set of DOCTYPEs recognized by browsers when selecting mode varies by browser. In practice, there is no reason to use a DOCTYPE other than <DOCTYPE html>
as defined in HTML5, though HTML5 also lists a few “legacy DOCTYPEs”. You can use that DOCTYPE if you want standards mode (recommended for new pages) and use no DOCTYPE if you want quirks mode (which you may need for legacy pages).
“Standards mode” generally means the mode of operation where a browser follows HTML, CSS, DOM and other specifications the best it can. It does not usually mean full conformance. “Quirks mode” is different in different browsers, but generally it means an attempt at imitating the behavior of very old browsers like IE 5. The purpose is to keep old pages working, under the assumption that they may rely on features and bugs in the old browsers. See the description What happens in Quirks Mode? Note that there is a rather different, more limited concept of “quirks mode” in HTML5, which closely resembles the document called Quirks Mode Living Standard.
A typical issue is that element widths are calculated differently in quirks mode and in standards mode. This means that the layout of a page may be more or less changed or even totally messed up, if a page designed to work in quirks mode is viewed in standards mode (or vice versa).
So you should use <!DOCTYPE html>
for new pages and keep whatever DOCTYPE (if any) you have been using for old pages.
However, quirks mode means, in some browsers, that many new features of CSS are not supported. This means that if you want to enhance an old page with some CSS3 feature, it may well be necessary to switch to a DOCTYPE that triggers standards mode. In such a case, you need to review and test the page to see whether it will run in standards mode.