How to return data from a Python SAX parser?

Question

I've been trying to parse some huge XML files that LXML won't grok, so I'm forced to parse them with xml.sax.

class SpamExtractor(sax.ContentHandler):
    def startElement(self, name, attrs):
        if name == "spam":
            print("We found a spam!")
            # now what?

The problem is that I don't understand how to actually return, or better, yield, the things that this handler finds to the caller, without waiting for the entire file to be parsed. So far, I've been messing around with threading.Thread and Queue.Queue, but that leads to all kinds of issues with threads that are really distracting me from the actual problem I'm trying to solve.

I know I could run the SAX parser in a separate process, but I feel there must be a simpler way to get the data out. Is there?

Fortnight

Answer 1

I thought I'd give this as another answer due to it being a completely different approach.

You might want to check out xml.etree.ElementTree.iterparse as it appears to do more what you want:

Parses an XML section into an element tree incrementally, and reports what’s going on to the user. source is a filename or file object containing XML data. events is a list of events to report back. If omitted, only “end” events are reported. parser is an optional parser instance. If not given, the standard XMLParser parser is used. Returns an iterator providing (event, elem) pairs.

You could then write a generator taking that iterator, doing what you want, and yielding the values you need.

e.g:

def find_spam(xml):
    for event, element in xml.etree.ElementTree.iterparse(xml):
        if element.tag == "spam":
            print("We found a spam!")
            # Potentially do something
            yield element

The difference is largely about what you want. ElementTree's iterator approach is more about collecting the data, while the SAX approach is more about acting upon it.

Belter

+1 but I'd add the following: (1) use cElementTree, not ElementTree (2) lxml also has an iterparse which provides the same or better functionality (3) you need to mention deleting nodes after you have extracted the required info (4) AFAICT (never tried it) a generator should work OK – Fatma

Screw SAX, I'm going with iterparse! Thanks heaps! – Fortnight

@JohnMachin I didn't know cElementTree existed - obviously, where speed is needed, it would be a good choice - but I don't see any reason to blanket suggest it's use where speed is not a high priority. <strike>As to deleting nodes, I don't see where that is needed, could you explain?</strike> - Explained seconds later by larsmans. – Belter

@JohnMachin, I guess you were referring to this in point (3)? – Fortnight

@larsmans No worries. I've always found ElementTree to be the nicest way to deal with XML - it makes a lot of sense to me, and seems very pythonic - it works easily, and the way you expect it to. – Belter

Yes, I'm used to lxml.etree myself. I'm going to use its version of iterparse now, seems to be blazing fast compared to xml.sax. – Fortnight

@Lattyware: about c?ElementTree: the functionality is about 99.99% identical, so the best recommendation is to use cElementTree unless you can't. – Fatma

@larsmans: Yes, iterparse builds a tree incrementally, you need to prune it incrementally. Check the lxml website (lxml.de/performance.html#parsing-and-serialising); it concedes that its iterparse is slower than cElementTree when parsing bytestrings (it's faster parsing unicode strings but nobody is going to decode before parsing!). For writing portable software, c?ElementTree is the way to go (it's included with Python; lxml may not even be obtainable (Jython, IronPython)). – Fatma

Answer 2

David Beazley demonstrates how to "yield" results from a sax ContentHandler using a coroutine:

cosax.py:

import xml.sax

class EventHandler(xml.sax.ContentHandler):
    def __init__(self,target):
        self.target = target
    def startElement(self,name,attrs):
        self.target.send(('start',(name,attrs._attrs)))
    def characters(self,text):
        self.target.send(('text',text))
    def endElement(self,name):
        self.target.send(('end',name))

def coroutine(func):
    def start(*args,**kwargs):
        cr = func(*args,**kwargs)
        cr.next()
        return cr
    return start

# example use
if __name__ == '__main__':
    @coroutine
    def printer():
        while True:
            event = (yield)
            print event

    xml.sax.parse("allroutes.xml",
                  EventHandler(printer()))

Above, every time self.target.send is called, the code inside printer runs starting from event = (yield). event is assigned to the arguments of self.target.send, and the code in printer is executed till the next (yield) is reached, sort of like how a generator works.

Whereas a generator is typically driven by a for-loop, the coroutine (e.g. printer) is driven by send calls.

Fernery

Answer 3

My understanding is the SAX parser is meant to do the work, not just pass data back up the food chain.

e.g:

class SpamExtractor(sax.ContentHandler):
    def __init__(self, canning_machine):
        self.canning_machine = canning_machine

    def startElement(self, name, attrs):
        if name == "spam":
            print("We found a spam!")
            self.canning_machine.can(name, attrs)

Belter

This is actually the setup that I have now, with a Queue replacing the canning_machine. But what I'd really is a way to stow the found items into a generator, avoiding threads. (Collecting the items in a list is not an option, the list won't fit in memory.) – Fortnight

My point is, where you have the loop going through the generator, move the stuff being done in the loop into the SAX parser. You can't create a generator as that would require changing what the xml.sax.parse function does. I agree, a generator would be a pythonic approach, but I believe you are limited by xml.sax. Please note I haven't done too much with SAX in Python, so I might be missing some great trick. – Belter

Answer 4

Basically there are three ways of parsing XML:

1. SAX-approach: it is an implementation of the Visitor pattern , the idea is that the events are pushed to your code.
2. StAX-approach: where you pull next element as long as you are ready (useful for partial parsing, i.e. reading only SOAP header)
3. DOM-approach, where you load everything into a tree in memory

You seem to need the second, but I am not sure that it is somewhere in the standard library.

Kirkuk

To clarify, for SAX those are not 'events' in the sense of async event/messages, those are just methods of the parent class you can implement and which will be called by it while it parses. You could use a SAX parser to send events to a queue. – Accessory