How to sign XML with xmlsec (or other more appropriate package)

Question

I start with an XML like this one:

myXML="""<soapenv:Envelope xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/" xmlns:mes="http://www.ercot.com/schema/2007-06/nodal/ews/message">
  <soapenv:Header> </soapenv:Header>
  <soapenv:Body>
  <RequestMessage xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns="http://www.ercot.com/schema/2007-06/nodal/ews/message">
    <Header>
      <Verb>get</Verb>
      <Noun>BidSet</Noun>
      <ReplayDetection>
        <Nonce>177766768</Nonce>
        <Created>2018-10-22T09:03:33.169-05:00</Created>
      </ReplayDetection>
      <Revision>1</Revision>
      <Source>QSAMP</Source>
      <UserID>USER1</UserID>
      <MessageID>test</MessageID>
      <Comment>test</Comment>
    </Header>
    <Request>
      <ID>QSAMP.20181020.EB.AB_C.BID123</ID>
    </Request>
  </RequestMessage>
 </soapenv:Body>
</soapenv:Envelope>"""

I need to sign it with so it looks like this

<soapenv:Envelope xmlns:mes="http://www.ercot.com/schema/2007-06/nodal/ews/message" xmlns:soapenv="http://schemas.xmlsoap.org/soap/envelope/">
  <soapenv:Header>
  <wsse:Security xmlns:wsse="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-secext-1.0.xsd" xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" soapenv:mustUnderstand="1">
    <wsse:BinarySecurityToken EncodingType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-soap-message-security-1.0#Base64Binary" ValueType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-x509-token-profile-1.0#X509v3" wsu:Id="X509-411BAD9927582E29B715402172715641">MIIEHTCCAwWgAwIBAgIBdDANBgkqhkiG9w0BAQsFADATMREwDwYDVQQKEwhFUkNPVERFVjAeFw0xNTA1MDYxODA4MjJaFw0yNTA1MDMxODA4MjJaMIIBHDELMAkGA1UEBhMCVVMxCzAJBgNVBAgTAlRYMQ8wDQYDVQQHEwZUYXlsb3IxETAPBgNVBAoTCEVSQ09UREVWMRswGQYDVQQLExJFbXBsb3llZUlEIC0gVVNFUjExFzAVBgNVBAsTDk1QIC0gMDQzMjI2ODg1MSQwIgYDVQQLExtEVU5TIE51bWJlciAtIDA0MzIyNjg4NTIwMDAxGjAYBgNVBAsTEUVSQ09UIERldmVsb3BtZW50MQ4wDAYDVQQLEwVVU0VSMTEWMBQGA1UECxMNMDQzMjI2ODg1MjAwMDEcMBoGA1UEAxMTMDQzMjI2ODg1MjAwMCBVU0VSMTEeMBwGCSqGSIb3DQEJARYPdXNlcjFAZXJjb3QuY29tMIIBIjANBgkqhkiG9w0BAQEFAAOCAQ8AMIIBCgKCAQEAwBBaWstqWoqqJb+xAU6KGEMt+OkQY6ubbEEQml5SxkqDQRppo8U1kMHqKn4XhP3llZQGsMIBBMelb3nE9R3cDRmVcm3gJrj+pdeRh0ZGgPNwkASI/ev6SGGwWvDgAPNr8l+/UrrzgAMaYe9DrDEiFFZt+8Si3bbF4TEPM8F+6lJ2sNPU/5QBSv99v3QtZrqZvctSs9HCEfuoU9f2jHFCW46PWhH1dJUiyH24GPjaw+ZbVhrz1ccJH+JwZKybXgUdL5/fzcOfWveMXkTd6ai0/pSHiJRrT8oOXuaBsprDi0dKYbZTNx9d+8/HOugmP1J235Zg/mogP9h5lmjbw6lm7wIDAQABo3EwbzAdBgNVHQ4EFgQUCYkq9iXtxcS2GnQWwzBVQrefV3UwQwYDVR0jBDwwOoAUxY2QrudEPe3ldnlEenhuy/ehYUShF6QVMBMxETAPBgNVBAoTCEVSQ09UREVWggkA4jYWHOuW0D4wCQYDVR0TBAIwADANBgkqhkiG9w0BAQsFAAOCAQEAqVrVOfUbYotkL9Qmq5C8VYae+msJt4aG5SfXeWtND58DhDhx32PbWsFcQHPvJhHR+jV9dBg5yHkYMt712DJUsm2Fec+Cgc7yiJIFq1Pbs0cpe2NQqVJEaSzPyd6KB/nbJonIRtPFE+F4w4EHM+mbhdWecx7Tk8MH9235suZX50+uWPnnlYp9uJnqtJ13Mb6KObaCw/b8wXtFD1lu4B5p2W1vfgwhRR3Zh/f5AhuqaS+ZuMrwY7eM5LM278L5mzC+uYMT4fzzszDA3koXNTEqcCYCytj/sTO/UVqIO57/kU1l/ea+yj5E+Ak9yGBBu0sjua66bC5XwNFEOVrQLBkiUg==</wsse:BinarySecurityToken>
    <ds:Signature xmlns:ds="http://www.w3.org/2000/09/xmldsig#" Id="SIG-411BAD9927582E29B715402172716115">
    <ds:SignedInfo>
    <ds:CanonicalizationMethod Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">
    <ec:InclusiveNamespaces xmlns:ec="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="mes soapenv"/>
    </ds:CanonicalizationMethod>
    <ds:SignatureMethod Algorithm="http://www.w3.org/2000/09/xmldsig#rsa-sha1"/>
    <ds:Reference URI="#id-411BAD9927582E29B715402172716114">
    <ds:Transforms>
    <ds:Transform Algorithm="http://www.w3.org/2001/10/xml-exc-c14n#">
    <ec:InclusiveNamespaces xmlns:ec="http://www.w3.org/2001/10/xml-exc-c14n#" PrefixList="mes"/>
    </ds:Transform>
    </ds:Transforms>
    <ds:DigestMethod Algorithm="http://www.w3.org/2000/09/xmldsig#sha1"/>
    <ds:DigestValue>Vd6yUSv013P7ov8AzF2IbYv7yS4=</ds:DigestValue>
    </ds:Reference>
    </ds:SignedInfo>
    <ds:SignatureValue>SnC9RHluvHxfg3zvfmoGHrfh6zfXSGUmGv9V351uhWgTn546tTU0/5LiaPsFEcfVxyWsoouVsBV9 VwCbw++6FmtehSCPH6CAO+1NngiE+miK6QThSqKJXj/5CbHwwfeQHqWRmf45AlCwvQiWhVqGi/tq
  YViFi5t0aIMrdhLJDRNUv17UNPKVjcowyIbKLKQxSqNxB/PED8tF0oHC7rRmsEr3x7NqO/VZBWZd OgCQggWiAdXiBy+SwoooAufMs6t+2+YOFQtWLOHuIx79X+hFi3Gqff1I5vfiHust7/rZdSzx1wB/
    T+aeNGIeIzQDNQoC55lhomgV0xp/3tZPHSzrqA==</ds:SignatureValue>
    <ds:KeyInfo Id="KI-411BAD9927582E29B715402172716112">
    <wsse:SecurityTokenReference wsu:Id="STR-411BAD9927582E29B715402172716113">
    <wsse:Reference URI="#X509-411BAD9927582E29B715402172715641" ValueType="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-x509-token-profile-1.0#X509v3"/>
    </wsse:SecurityTokenReference>
    </ds:KeyInfo>
    </ds:Signature>
    </wsse:Security>
    </soapenv:Header>
    <soapenv:Body xmlns:wsu="http://docs.oasis-open.org/wss/2004/01/oasis-200401-wss-wssecurity-utility-1.0.xsd" wsu:Id="id-411BAD9927582E29B715402172716114">
    <RequestMessage xmlns="http://www.ercot.com/schema/2007-06/nodal/ews/message" xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
    <Header>
    <Verb>get</Verb>
    <Noun>BidSet</Noun>
    <ReplayDetection>
    <Nonce>177766768</Nonce>
    <Created>2018-10-22T09:07:33.169-05:00</Created>
    </ReplayDetection>
    <Revision>1</Revision>
    <Source>QSAMP</Source>
    <UserID>USER1</UserID>
    <MessageID>test</MessageID>
    <Comment>test</Comment>
    </Header>
    <Request>
    <ID>QSAMP.20181020.EB.AB_C.BID123</ID>
    </Request>
    </RequestMessage>
    </soapenv:Body>
    </soapenv:Envelope>

As an aside, the certificate information in there isn't real so no worries there.

I looked at the example page from the doc but I don't know what their source xml looks like so it makes molding my use case to it pretty tricky.

Douceur

You start reading examples.html#sign – Maundy

@Maundy I got an error at ctx.sign. I'm not at computer now but it wasn't a descriptive error. – Douceur

Answer 1

It took some fumbling around the source of zeep, xmlsec, and even trying out lxml constructors, but here it is:

from zeep.wsse.signature import sign_envelope
from lxml import etree

raw_xml = open('unsigned-soapenv.xml').read()
xml_root_element = etree.fromstring(raw_xml)

signed = sign_envelope(
    xml_root_element,
    'rsakey.pem',
    'rsacert.pem'
)

tree = etree.ElementTree(xml_root_element)
tree.write('signed-soapenv.xml')

unsigned-soapenv.xml is your envelope.

If you need the sample signature and certificate, they are from python-xmlsec's test data.

What was invaluable was ipdb, which is a debugger with auto-complete. Try it like this, in your terminal: import ipdb; ipdb.set_trace()

Hargrave

Can you also submit this example to zeep issue tracker? – Lucrece

@plaes: I just created a pull request to slightly improve the interface, and document it: github.com/mvantellingen/python-zeep/pull/858 - thanks for the suggestion! – Hargrave

@plaes: it seems that the maintainer was not too involved with zeep in the past year; I wouldn't count on it being accepted too soon. – Hargrave

Thanks you for your effort, it is much appreciated. However, if you're willing to dig more, there are a couple issues unfortunately. It might just be that these need to be cleaned up manually after sign_envelope but maybe there's a hidden parameter for these: This output is missing the BinarySecurityToken tag, there are two places where there should be an InclusivesNamespaces tag. There shouldn't be a X509Data section but should be a Reference tag. There shouldn't be the Timestamp tag added at the end. – Douceur

@DeanMacGregor: Indeed, it looks like the implementation of BinarySecurityToken is completely missing. There is a reference however, to an obsolete project; and here is where it generates such a token: github.com/mvantellingen/py-soap-wsse/blob/… – Hargrave

About InclusiveNamespaces, X509Data and Reference tags, I can't help you, and couldn't find any useful code; they are over my head. You can look through the implementation of sign_envelope and perhaps customize it to your needs (and the standard): github.com/mvantellingen/python-zeep/blob/… – Hargrave

@DeanMacGregor How did you end up signing the SOAP envelope? – Manis

Never got it to work – Douceur

Answer 2

python-zeep package has preliminary support for WSSE - check out the documentation.

Lucrece

I've stumbled on zeep and suds and neither make the solution obvious. – Douceur

Thank you for the reference to zeep. I used it in my answer. – Hargrave