Suds generates empty elements; how to remove them?

Question

[Major Edit based on experience since 1st post two days ago.]

I am building a Python SOAP/XML script using Suds, but am struggling to get the code to generate SOAP/XML that is acceptable to the server. I had thought that the issue was that Suds was not generating prefixes for inner elements, but subsequently it turns out that the lack of prefixes (see Sh-Data and inner elements) is not an issue, as the Sh-Data and MetaSwitchData elements declare appropriate namespaces (see below).

<SOAP-ENV:Envelope xmlns:ns3="http://www.metaswitch.com/ems/soap/sh" xmlns:ns0="http://www.metaswitch.com/ems/soap/sh/userdata" xmlns:ns1="http://www.metaswitch.com/ems/soap/sh/servicedata" xmlns:ns2="http://schemas.xmlsoap.org/soap/envelope/" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xmlns:SOAP-ENV="http://schemas.xmlsoap.org/soap/envelope/">
   <SOAP-ENV:Header/>
   <ns2:Body>
      <ns3:ShUpdate>
         <ns3:UserIdentity>Meribel/TD Test Sub Gateway 3</ns3:UserIdentity>
         <ns3:DataReference>0</ns3:DataReference>
         <ns3:UserData>
            <Sh-Data xmlns="http://www.metaswitch.com/ems/soap/sh/userdata">
               <RepositoryData>
                  <ServiceIndication>Meta_SubG_BaseInformation</ServiceIndication>
                  <SequenceNumber>0</SequenceNumber>
                  <ServiceData>
                     <MetaSwitchData xmlns="http://www.metaswitch.com/ems/soap/sh/servicedata" IgnoreSequenceNumber="False" MetaSwitchVersion="?">
                        <Meta_SubG_BaseInformation Action="apply">
                           <NetworkElementName>Meribel</NetworkElementName>
                           <Description>TD Test Sub Gateway 3</Description>
                           <DomainName>test.datcon.co.uk</DomainName>
                           <MediaGatewayModel>Cisco ATA</MediaGatewayModel>
                           <CallFeatureServerControlStatus/>
                           <CallAgentControlStatus/>
                           <UseStaticNATMapping/>
                           <AuthenticationRequired/>
                           <ProviderStatus/>
                           <DeactivationMode/>
                        </Meta_SubG_BaseInformation>
                     </MetaSwitchData>
                  </ServiceData>
               </RepositoryData>
            </Sh-Data>
         </ns3:UserData>
         <ns3:OriginHost>[email protected]?clientVersion=7.3</ns3:OriginHost>
      </ns3:ShUpdate>
   </ns2:Body>
</SOAP-ENV:Envelope>

But this still fails. The issue is that Suds generates empty elements for optional elements (marked as Mandatory = No in the WSDL). But the server requires that an optional element is either present with a sensible value or absent, and I get the following error (because the <CallFeatureServerControlStatus/> element is not one of the allowable values.

The user data provided did not validate against the MetaSwitch XML Schema for user data.
Details: cvc-enumeration-valid: Value '' is not facet-valid with respect to enumeration '[Controlling, Abandoned, Cautiously controlling]'. It must be a value from the enumeration.

If I take the generated SOAP/XML into SOAPUI and delete the empty elements, the request works just fine.

Is there a way to get Suds to either not generate empty elements for optional fields, or for me to remove them in code afterwards?

Major Update

I have solved this problem (which I've seen elsewhere) but in a pretty inelegant way. So I am posting my current solution in the hope that a) it helps others and/or b) someone can suggest a better work-around.

It turns out that the problem was not that Suds generates empty elements for optional elements (marked as Mandatory = No in the WSDL). But rather that that Suds generates empty elements for optional complex elements. For example the following Meta_SubG_BaseInformation elements are simple elements and Suds does not generate anything for them in the SOAP/XML.

<xs:element name="CMTS" type="xs:string" minOccurs="0">
    <xs:annotation>
        <xs:documentation>
            <d:DisplayName firstVersion="5.0" lastVersion="7.4">CMTS</d:DisplayName>
            <d:ValidFrom>5.0</d:ValidFrom>
            <d:ValidTo>7.4</d:ValidTo>
            <d:Type firstVersion="5.0" lastVersion="7.4">String</d:Type>
            <d:BaseAccess firstVersion="5.0" lastVersion="7.4">RWRWRW</d:BaseAccess>
            <d:Mandatory firstVersion="5.0" lastVersion="7.4">No</d:Mandatory>
            <d:MaxLength firstVersion="5.0" lastVersion="7.4">1024</d:MaxLength>
        </xs:documentation>
    </xs:annotation>
</xs:element>

<xs:element name="TAGLocation" type="xs:string" minOccurs="0">
    <xs:annotation>
        <xs:documentation>
            <d:DisplayName>Preferred location of Trunk Gateway</d:DisplayName>
            <d:Type>String</d:Type>
            <d:BaseAccess>RWRWRW</d:BaseAccess>
            <d:Mandatory>No</d:Mandatory>
            <d:DefaultValue>None</d:DefaultValue>
            <d:MaxLength>1024</d:MaxLength>
        </xs:documentation>
    </xs:annotation>
</xs:element>

In contrast the following Meta_SubG_BaseInformation element is a complex element, and even when it is optional and my code does not assign a value to it, it ends up in the generated SOAP/XML.

<xs:element name="ProviderStatus" type="tMeta_SubG_BaseInformation_ProviderStatus" minOccurs="0">
    <xs:annotation>
        <xs:documentation>
            <d:DisplayName>Provider status</d:DisplayName>
            <d:Type>Choice of values</d:Type>
            <d:BaseAccess>R-R-R-</d:BaseAccess>
            <d:Mandatory>No</d:Mandatory>
            <d:Values>
                <d:Value>Unavailable</d:Value>
                <d:Value>Available</d:Value>
                <d:Value>Inactive</d:Value>
                <d:Value>Active</d:Value>
                <d:Value>Out of service</d:Value>
                <d:Value>Quiescing</d:Value>
                <d:Value>Unconfigured</d:Value>
                <d:Value>Pending available</d:Value>
            </d:Values>
        </xs:documentation>
    </xs:annotation>
</xs:element>

Suds generates the following for ProviderStatus which (as stated above) upsets my server.

<ProviderStatus/>

The work-around is to set all Meta_SubG_BaseInformation elements to None after creating the parent element, and before assigning values, as in the following. This is superfluous for the simple elements, but does ensure that non-assigned complex elements do not result in generated SOAP/XML.

subGatewayBaseInformation = client.factory.create('ns1:Meta_SubG_BaseInformation')
for (el) in subGatewayBaseInformation:
  subGatewayBaseInformation.__setitem__(el[0], None)
subGatewayBaseInformation._Action            = 'apply'
subGatewayBaseInformation.NetworkElementName = 'Meribel'
etc...

This results in Suds generating SOAP/XML without empty elements, which is acceptable to my server.

But does anyone know of a cleaner way to achieve the same effect?

Solution below is based on answers / comments from both dusan and Roland Smith below.

This solution uses a Suds MessagePlugin to prune "empty" XML of the form <SubscriberType/> before Suds puts the request on the wire. We only need to prune on ShUpdates (where we are updating data on the server), and the logic (especially the indexing down into the children to get the service indication element list) is very specific to the WSDL. It would not work for different WSDL.

class MyPlugin(MessagePlugin):
  def marshalled(self, context):
    pruned = []
    req = context.envelope.children[1].children[0]
    if (req.name == 'ShUpdate'):
      si = req.children[2].children[0].children[0].children[2].children[0].children[0]
      for el in si.children:
        if re.match('<[a-zA-Z0-9]*/>', Element.plain(el)):
          pruned.append(el)
      for p in pruned:
        si.children.remove(p)

And then we just need to reference the plugin when we create the client.

client = Client(url, plugins=[MyPlugin()])

Mongolian

you can look at the following links : 1> #2470488 2>#2965367. – Pious

I saw those. They are to do with adding headers (unless I'm mis-understanding / mis-reading hem) as opposed to getting Suds to include prefixes for namespaces that it does know about). – Mongolian

hoping someone can add a cleaner answer! – Macneil

Answer 1

You can filter the empty elements out with a regular expression.

Assuming your XML data is in the string xmltext;

import re
filteredtext = re.sub('\s+<.*?/>', '', xmltext)

Haimes

Good try, but sadly that doesn't work. After reading the WSDL, Suds does not directly expose the XML. So the Python code never has access to a string representation of the XML. Instead Suds provides access to a series of object types, where the Python code can set values etc. And then Suds internally generates / sends the XML when the Python code invokes a method defined in the XML. – Mongolian

If I'm reading the docs correctly, the current suds version supports plugins. It seems that deriving a class from MessagePlugin and re-definging the sending() method, you can inspect/modify the text before it was sent. Overriding the receiving() method allows you to inspect/modify received XML before it is parsed. Combined with the regex, that might do the trick. – Haimes

It appears that there is an issue with the sending MessagePLugin, so that even if I get the regex correct (tested offline), when the the SOAP request hits the wire it contains the original XML, not the sub'd XML. See lists.fedoraproject.org/pipermail/suds/2010-November/…. – Mongolian

Added a solution based on a mix of both dusan and Roland Smith anserws/comments. – Mongolian

Answer 2

You can use a plugin to modify the XML before is sent to the server (my answer is based on Ronald Smith's solution):

from suds.plugin import MessagePlugin
from suds.client import Client
import re

class MyPlugin(MessagePlugin):
    def sending(self, context):
        context.envelope = re.sub('\s+<.*?/>', '', context.envelope)


client = Client(URL_WSDL, plugins=[MyPlugin()])

Citing the documentation:

The MessagePlugin currently has (5) hooks ::
(...)
sending()
Provides the plugin with the opportunity to inspect/modify the message text before it is sent.

Basically Suds will call sending before the XML is sent, so you can modify the generated XML (contained in context.envelope). You have to pass the plugin class MyPlugin to the Client constructor for this to work.

Edit

Another way is to use marshalled to modify the XML structure, removing the empty elements (untested code):

class MyPlugin(MessagePlugin):
    def marshalled(self, context):
        #remove empty tags inside the Body element
        #context.envelope[0] is the SOAP-ENV:Header element
        context.envelope[1].prune()

Tuneberg

Interesting. I didn't know about MessagePlugins. So... I've tested the approach, and modifying the XML in a MessagePlugin gets me close - but getting the regex syntax right is challenging. In particular it has to leave <SOAP-ENV:Header/> alone while stripping out (for example) <SubscriberType/> and <TeenServiceLine/>. Are you guys better at regex than me? – Mongolian

Using the marshalled MessagePlugin looks promising, but I can't make assumptions about the position of elements to be pruned. [There are are several thousand service indications defined in the WSDL, with almost as many unique formats / fields.] I tried doing a tostring() on the Elements comprising the context (for ct in context.envelope: ctStr = el.tostring(ct)) after importing xml.etree.ElementTree as el, but got an error AttributeError: Element instance has no attribute 'getiterator'? – Mongolian

Try this: for child in context.envelope.children: print child – Tuneberg

I was doing for ct in context.envelope rather than for ct in context.envelope.children but both return ct as an Element, which does not convert to string (as per above) and therefore I can't compare or do regex substitutions on individual elements. – Mongolian

You can take inspiration from prune implementation: fedorahosted.org/suds/browser/trunk/suds/sax/element.py#L878 . Use children() to navigate down the XML and remove to delete the empty elements. – Tuneberg

@Mongolian Try the following regex \s+<[^:]*?/>. – Haimes

I got it working using elements from both dusan and Roland Smith. See my answer in the latest update to my own question above. – Mongolian

Answer 3

There's an even easier way - no need for any Reg Ex or exciting iterators ;)

First, define the plugin:

class PrunePlugin(MessagePlugin):
    def marshalled(self, context):
        context.envelope = context.envelope.prune()

Then use it when creating the client:

client = Client(url, plugins=[PrunePlugin()])

The prune() method will remove any empty nodes, as documented here: http://jortel.fedorapeople.org/suds/doc/suds.sax.element.Element-class.html

Psalterium

This is nice and clean. If you don't have control over the client creation but have access to it after the fact, you can set the plugin option like this: client.set_options(plugins=[PrunePlugin()]) – Baccivorous

Answer 4

The Suds factory method generates a regular Python object with regular python attributes that map to the WSDL type definition.

You can use the 'del' builtin function to remove attributes.

>>> order_details = c.factory.create('ns2:OrderDetails')
>>> order_details
(OrderDetails){
   Amount = None
   CurrencyCode = None
   OrderChannelType =
      (OrderChannelType){
         value = None
      }
   OrderDeliveryType =
      (OrderDeliveryType){
         value = None
      }
   OrderLines =
      (ArrayOfOrderLine){
         OrderLine[] = <empty>
      }
   OrderNo = None
   TotalOrderValue = None
 }
>>> del order_details.OrderLines
>>> del order_details.OrderDeliveryType
>>> del order_details.OrderChannelType
>>> order_details
(OrderDetails){
   Amount = None
   CurrencyCode = None
   OrderNo = None
   TotalOrderValue = None
 }

Elviraelvis

Answer 5

What do you think of the following MonkeyPatch to skip complex types with a None value?

from suds.mx.literal import Typed
old_skip = Typed.skip
def new_skip(self, content):
    x = old_skip(self, content)
    if not x and getattr(content.value, 'value', False) is None:
        x = True
    return x
Typed.skip = new_skip

Mcduffie

can you adjust this code so its ready to be used as a plugin? – Audacious

Answer 6

I know this one was closed a long time ago, but after working on the problem personally I find the current answers lacking.

Using the sending method on the MessagePlugin won't work, because despite what the documentation heavily implies, you cannot actually change the message string from there. You can only retrieve the final result.

The marshalled method, as previously mentioned, is best for this, as it does allow you to affect the XML. I created the following plugin to fix the issue for myself:

class ClearEmpty(MessagePlugin):
    def clear_empty_tags(self, tags):
        for tag in tags:
            children = tag.getChildren()[:]
            if children:
                self.clear_empty_tags(children)
            if re.match(r'^<[^>]+?/>$', tag.plain()):
                tag.parent.remove(tag)

    def marshalled(self, context):
        self.clear_empty_tags(context.envelope.getChildren()[:])

This will eliminate all empty tags. You can tailor this as needed if you only need to remove some empty tags from some place, but this recursive function works and (unless your XML schema is so unspeakably bad as to have nesting greater than the call depth of Python), shouldn't cause a problem. Note that we copy the lists here because using remove() mangles them as we're iterating and causes problems.

On an additional note, the regex that has been given by other answers is bad-- \s+<.*?/> used on <test> <thingus/> </test> will match <test> <thingus/>, and not just <thingus/> as you might expect. This is because > is considered 'any character' by .. If you really need to use regex to fix this problem on rendered XML (Note: XML is a complex syntax that is better handled by a lexer), the correct one would be <[^>]*/>.

We use it here because I could not figure out the most correct way to ask the lexer 'is this a stand alone empty tag', other than to examine that tag's rendered output and regex against that. In such case I also added the ^ and $ tokens because rendering the tag in this method renders its entire context, and so that means that any blank tag underneath a particular tag would be matched. We just want the one particular tag to be matched so we can tell the API to remove it from the tree.

Finally, to help those searching for what might have prompted this question in the first place, the issue came up for me when I received an error message like this:

cvc-enumeration-valid: Value '' is not facet-valid with respect to enumeration

This is because the empty tag causes the server to interpret everything that would be under that tag as null values/empty strings.

Lederer

Answer 7

I thought I'd share a pretty simple update on the solution above that should work for any WSDL: Note that the sending method is not necessary - it's so that you can audit your changes, as the Client's debug request printing fires before the marshal method runs.

class XMLBS_Plugin(MessagePlugin):
def marshalled(self, context):
    def w(x):
        if x.isempty():
            print "EMPTY: ", x
            x.detach()

    context.envelope.walk(w)

def sending(self,context):
    c = copy.deepcopy(context.envelope)
    c=c.replace('><','>\n<') # some sort of readability
    logging.info("SENDING: \n%s"%c)

Chapen

By walk(f) do you mean walk(w)? – Tuneberg

yep - the markup on SO changed that function's name as I was previewing it. Very confusing - I'll edit. – Taphouse