Is it possible to parse sub-trees with Groovy XMLSlurper

Question

Does anyone know whether it is possible to utilise XMLSlurper in a fashion that means individual sub-trees can be pulled from a very large XML document and processed individually?

Imagine you've got a huge XML feed containing a root element that has thousands of direct child elements that you can process individually. Obviously, reading the whole document into memory is a no-no but, as each child of the root is itself modestly sized, it would be nice to stream through the document but apply XMLSlurper niceness to each of the child elements in turn. As each child element is processed, garbage collection can clean up memory used to process it. In this way we get the great ease of XMLSlurper (such concise syntax) with the low memory footprint of streaming (e.g. SAX).

I'd be interested to know if anyone has ideas on this and/or whether you've come across this requirement yourselves.

Villalobos

Answer 1

Initializing an XmlSlurper instance means, calling one of its overloaded parse(..) methods (or the parseText(String) method). Upon this call, XmlSlurper will (use SAX events, at least, to) construct an in-memory GPathResult that holds the complete information on the XML elements and attributes, and their structure.

So, no, the XmlSlurper does not provide an API to parse XML document portions, only.

What can be done is, extending XmlSlurper, overwriting the parse*(..) methods, pre-processing the XML by using a custom SAX handler, gathering the desired portions of XML, and forwarding these to one of the XmlSlurper.parse*(..) methods.

Whisker

Thanks - I'd come to the conclusion that I'd have to do something like this. However, it does mean I'd essentially have to do a double-pass and thus it does affect performance. – Villalobos

Answer 2

You can use StAX API together with XmlSlurper to parse subtrees.

// Example of using StAX to split a large XML document and parse a single element using XmlSlurper

import javax.xml.stream.XMLInputFactory
import javax.xml.stream.XMLStreamReader
import javax.xml.transform.Transformer
import javax.xml.transform.TransformerFactory
import javax.xml.transform.sax.SAXResult
import javax.xml.transform.stax.StAXSource

def url = new URL("http://repo2.maven.org/maven2/archetype-catalog.xml")
url.withInputStream { inputStream ->
    def xmlStreamReader = XMLInputFactory.newInstance().createXMLStreamReader(inputStream)
    def transformer = TransformerFactory.newInstance().newTransformer()
    while (xmlStreamReader.hasNext()) {
        xmlStreamReader.next()
        if (xmlStreamReader.isStartElement() && xmlStreamReader.getLocalName() == 'archetype') {
            // Example of splitting a large XML document and parsing a single element with XmlSlurper at a time
            def xmlSlurper = new XmlSlurper()
            transformer.transform(new StAXSource(xmlStreamReader), new SAXResult(xmlSlurper))
            def archetype = xmlSlurper.document
            println "${archetype.groupId} ${archetype.artifactId} ${archetype.version}"
        }
    }
}

Descent

I answered this on Twitter a while ago twitter.com/lhotari/status/694002023562416128 so I decided to add the answer also to SO. – Descent