Basically the difference between domain and work group mode is not defined by the value of the registry flag "workgroup" mentioned by @engin. This flag just reflects current operational mode but doesn't set it.
Whether you run in domain or workgroup mode is defined whether you installed MSMQ on domain controller or on a member server. Details about differences between these two modes can be found here: https://support.microsoft.com/en-us/kb/884974/
MSMQ 1.0 used to support domain mode only. Current MSMQ version is 5.0.
Next you may see quite interesting behavior when you installed MSMQ on your DC, your workgroup flag continuously reverts to 1 after each MSMQ service restart. This means that you have to grant Network Service account the Create MSMQ Configuration Objects permission to the computer object in Active Directory Domain Services before installing the Directory Services Integration feature on a computer that is a domain controller.
You may find details on how to do it here:
https://technet.microsoft.com/en-us/library/cc730960.aspx
MSMQ runs under the (less privileged) Network Service account instead of (all powerful) Local System account starting from version 4.0 (Vista/Server 2008)
So to answer @mkus question more directly to "set" domain mode you just install MSMQ on domain controller and make sure that proper permissions in place for Network Service account. Once this is done you well see it operating in domain mode with workgroup flag switched to 0 automatically to reflect this.
Also couple of links to clarify issues around MSMQ objects permissions and when/why you need to set them:
http://blogs.msdn.com/b/johnbreakwell/archive/2009/08/03/default-msmq-queue-permissions-have-changed-in-msmq-4-0.aspx. In short starting from MSMQ 4.0 Everyone and Anonymous Logon were removed from default MSMQ objects ACLs as precaution against DoS attacks (though there are exlusions to this change and Workgroup mode is one of those).
And as you may read in Technet article below you need to go a grant certain rigts to MSMQ objects either to Network Service OR to Computer accounts when installing the Routing Service feature on a Windows Server 2008 R2 (or later) computer that is not a domain controller OR when installing the Directory Service Integration feature of Message Queuing on a Windows Server 2008 R2 (or later) computer that is a domain controller. See details here:
https://technet.microsoft.com/en-us/library/cc749102(v=ws.10).aspx