FWIW the Wikipedia epub article states
When present, DRMed EPUB files must contain a file called rights.xml within the META-INF directory at the root level of the ZIP container.
epubcheck as of v1.0 is reported to detect DRM-protection via the existence of encryption.xml and report an error. The epub3 spec says
An optional encryption.xml file within the META-INF directory at the
root level of the container file system holds all encryption
information on the contents of the container. This file is an XML
document whose root element is encryption. The encryption element
contains child elements of type EncryptedKey and EncryptedData as
defined by [XML ENC Core]. Each EncryptedData element describes how
one or more files within the container are encrypted. Consequently, if
any resource within the container is encrypted, encryption.xml must be
present to indicate that the resource is encrypted and provide
information on how it is encrypted.
The EPUB3 is vague on rights.xml, its meaning and whether it is mandatory for DRM'd files, so it might be best not to depend on it.