Quoting from the Extensibility section of the HTML5 specification:
For markup-level features that can be limited to the XML serialization and need not be supported in the HTML serialization, vendors should use the namespace mechanism to define custom namespaces in which the non-standard elements and attributes are supported.
So if you're using the XML serialization of HTML5, its legal for you to do something like this:
<greeting xmlns="http://example.com/customNamespace">Hello!</greeting>
However, if you're using the HTML syntax you are much more limited in what you can do.
For markup-level features that are intended for use with the HTML syntax, extensions should be limited to new attributes of the form "x-vendor-feature" [...] New element names should not be created.
But those instructions are primarily directed at browser vendors, who would assumedly be providing visual styling and functionality for whatever custom elements they chose to create.
For an author, though, while it may be legal to embed a custom element in the page (at least in the XML serialization), you're not going to get anything more than a node in the DOM. If you want your custom element to actually do something, or be rendered in some special way, you should be looking at the Custom Elements specification.
For a more gentle primer on the subject, read the Web Components Introduction, which also includes information about the Shadow DOM and other related specifications. These specs are still working drafts at the moment - you can see the current status here - but they are being actively developed.
As an example, a simple definition for a greeting
element might look something like this:
<element name="greeting">
<template>
<style scoped>
span { color:gray; }
</style>
<span>Simon says:</span>
<q><content/></q>
</template>
</element>
This tells the browser to render the element content in quotes, and prefixed by the text "Simon says:" which is styled with the color gray. Typically a custom element definition like this would be stored in a separate html file that you would import with a link.
<link rel="import" href="greeting-definition.html" />
Although you can also include it inline if you want.
I've created a working demonstration of the above definition using the Polymer polyfill library which you can see here. Note that this is using an old version of the Polymer library - more recent versions work quite differently. However, with the spec still in development, this is not something I would recommend using in production code anyway.