Parsing Highly Nested XML without DOM in Java

Question

I've been tasked with fixing a rather irritating Heap out of memory issue. IBM offers a Cognos SDK that we use with Java, and we query all of the packages stored on a content store, which are returned in an xml format. Then we parse that xml and write it to a sql database. Profiling reveals that the worst memory issues are caused by Char[], which isn't very helpful (and the heaps are so large it's hard to profile), but does point towards the DOM parser.

We're talking 500-1500 xml files (well, technically, XML text streams) that are absurdly deeply nested and vary in size and occasionally in structure. Size varies from a few KB up to 30 MB in size, and the program will eat upwards of 8 GB of memory after about 300 packages. Programmer before me handled this by doing a manual System.gc call after every xml parse, which I wish to move away from (and it also doesn't actually solve the issue, just makes it viable on the smallest, 500 package server).

I tried to use JAXB, but it has an odd structure that made it very difficult to use here (it has some "folder or querySubject" thing going on). I tried STAX for several hours last week, but wasn't able to quite get working, same for WoodStox. I couldn't really find examples or tutorials on doing this for either. JDOM was what I examined next (as I've read that it has significantly better memory handling than pure DOM), but I can't figure out how to get it to parse quite as deeply as DOM. Current DOM parsing :

            is = new ByteArrayInputStream(xml.getBytes("UTF-8"));
            xmlDoc = builder.parse(is);
            is.close();
        String _path, datatype, regularAggregate, description, formula;
        String table, tableLoc;

            NodeList elements = xmlDoc.getElementsByTagName("*");
            for (int j = 0; j < elements.getLength(); j++) {


                Element element = (Element) elements.item(j);
                String nodeName = element.getNodeName();
                if (nodeName=="queryItem" || nodeName=="measure"|| 
                nodeName=="calculation" || nodeName=="filter") {
                    if (element.hasAttribute("_path")) {
                    path = element.getAttribute("_path"));
                    } 

and so on for each attribute

My JDOM attempt. Currently, it only prints the root element, and I've yet to be able to go deeper than the first child layer :

SAXBuilder saxBuilder = new SAXBuilder();
Document document = saxBuilder.build(inputFile);

System.out.println("Root element :" + document.getRootElement().getName());
Element root = document.getRootElement();

List<Element> rList = root.getChildren("folder");

if (rList!= null) {
    for (Element node : rList) {
        List<Element> elements = node.getChildren("queryItem");
        if (elements!=null) {
            for (Element a:elements) {
            System.out.println(a.getAttribute("_path"));    
            }
            elements.size();
            rList.removeAll(elements);

        }
    }

Generated xsd structure of a random package:

<?xml version="1.0" encoding="UTF-8"?>
<xs:schema xmlns:xs="http://www.w3.org/2001/XMLSchema" elementFormDefault="qualified">
  <xs:element name="ResponseRoot">
    <xs:complexType>
      <xs:sequence>
        <xs:element ref="folder"/>
        <xs:element ref="package"/>
      </xs:sequence>
    </xs:complexType>
  </xs:element>
  <xs:element name="package">
    <xs:complexType>
      <xs:attribute name="description" use="required"/>
      <xs:attribute name="name" use="required"/>
      <xs:attribute name="screenTip" use="required"/>
    </xs:complexType>
  </xs:element>
  <xs:element name="folder">
    <xs:complexType>
      <xs:sequence>
        <xs:choice minOccurs="0" maxOccurs="unbounded">
          <xs:element ref="folder"/>
          <xs:element ref="querySubject"/>
        </xs:choice>
        <xs:element minOccurs="0" maxOccurs="unbounded" ref="filter"/>
      </xs:sequence>
      <xs:attribute name="_path" use="required"/>
      <xs:attribute name="_ref" use="required"/>
      <xs:attribute name="description" use="required"/>
      <xs:attribute name="isNamespace" use="required" type="xs:integer"/>
      <xs:attribute name="name" use="required"/>
      <xs:attribute name="screenTip" use="required"/>
    </xs:complexType>
  </xs:element>
  <xs:element name="querySubject">
    <xs:complexType>
      <xs:sequence>
        <xs:element minOccurs="0" maxOccurs="unbounded" ref="queryItem"/>
        <xs:element minOccurs="0" maxOccurs="unbounded" ref="queryItemFolder"/>
      </xs:sequence>
      <xs:attribute name="_path" use="required"/>
      <xs:attribute name="_ref" use="required"/>
      <xs:attribute name="description" use="required"/>
      <xs:attribute name="name" use="required"/>
      <xs:attribute name="screenTip" use="required"/>
    </xs:complexType>
  </xs:element>
  <xs:element name="filter">
    <xs:complexType>
      <xs:attribute name="_path" use="required"/>
      <xs:attribute name="_ref" use="required"/>
      <xs:attribute name="description" use="required"/>
      <xs:attribute name="expression" use="required"/>
      <xs:attribute name="name" use="required"/>
      <xs:attribute name="screenTip" use="required"/>
    </xs:complexType>
  </xs:element>
  <xs:element name="queryItem">
    <xs:complexType>
      <xs:attribute name="_path" use="required"/>
      <xs:attribute name="_ref" use="required"/>
      <xs:attribute name="currency" use="required"/>
      <xs:attribute name="datatype" use="required" type="xs:NCName"/>
      <xs:attribute name="description" use="required"/>
      <xs:attribute name="displayType" use="required" type="xs:NCName"/>
      <xs:attribute name="expression" use="required"/>
      <xs:attribute name="name" use="required"/>
      <xs:attribute name="promptCascadeOnRef" use="required"/>
      <xs:attribute name="promptDisplayItemRef" use="required"/>
      <xs:attribute name="promptFilterItemRef" use="required"/>
      <xs:attribute name="promptType" use="required" type="xs:NCName"/>
      <xs:attribute name="regularAggregate" use="required" type="xs:NCName"/>
      <xs:attribute name="screenTip" use="required"/>
      <xs:attribute name="unSortable" use="required" type="xs:integer"/>
      <xs:attribute name="usage" use="required" type="xs:NCName"/>
    </xs:complexType>
  </xs:element>
  xs:element name="queryItemFolder">
    <xs:complexType>
      <xs:choice minOccurs="0" maxOccurs="unbounded">
        <xs:element ref="queryItem"/>
        <xs:element ref="queryItemFolder"/>
      </xs:choice>
      <xs:attribute name="_path" use="required"/>
      <xs:attribute name="_ref" use="required"/>
      <xs:attribute name="description" use="required"/>
      <xs:attribute name="name" use="required"/>
      <xs:attribute name="screenTip" use="required"/>
     </xs:complexType>
     </xs:element>
     </xs:schema>

Ferbam

For sufficiently large and complex XML files with deep root structures or just enormous file depth, you need to stream and parse it on the fly using SAX. Trying to store everything as a POJO will be inefficient and has no guarantee of succeeding if the file is sufficiently bloated. You'd need something like this. tutorialspoint.com/java_xml/java_sax_parser.htm – Vernation

Do you have an example of SAX with nested structures? The ones on tutorialspoint are quite simple, and the only one I could find online was rather dated : mohammed-technical.blogspot.com/2010/08/… Has SAX changed in the past 8 years? It's somewhat intimidating in its complexity – Ferbam

Don't use SAX. Use StAX, as it's much easier to use, and is equally low on memory footprint. – Commode

tutorialspoint.com/java_xml/java_stax_parse_document.htm This would be StAX. Concept is similar to old SAX but easier. You'd basically have to define each structure yourself, but the speed of testing and debugging is very high. – Vernation

Answer 1

For nested structures, it's easiest to manage if you create a method for each element type.

Example

public static void main(String[] args) throws Exception {
    String xml = "<root>" +
                   "<folder name=\"A\">" +
                     "<folder name=\"B\">" +
                       "<book name=\"Learn Java\">" +
                         "<chapter name=\"Hello, World!\"/>" +
                         "<chapter name=\"Variables and Types\"/>" +
                       "</book>" +
                     "</folder>" +
                   "</folder>" +
                 "</root>";
    XMLInputFactory factory = XMLInputFactory.newFactory();
    XMLStreamReader reader = factory.createXMLStreamReader(new StringReader(xml));
    try {
        reader.nextTag(); // Position on root element
        String tagName = reader.getLocalName();
        if (! tagName.equals("root"))
            throw new XMLStreamException("Expected <root> element, found: " + tagName, reader.getLocation());
        parseRoot(reader);
    } finally {
        reader.close();
    }
}

private static void parseRoot(XMLStreamReader reader) throws XMLStreamException {
    while (reader.nextTag() != XMLStreamConstants.END_ELEMENT) {
        String tagName = reader.getLocalName();
        if (tagName.equals("folder")) {
            parseFolder(reader, Collections.emptyList());
        } else {
            throw new XMLStreamException("Expected <folder> element, found: " + tagName, reader.getLocation());
        }
    }
}

private static void parseFolder(XMLStreamReader reader, List<String> parentPaths) throws XMLStreamException {
    String folderName = reader.getAttributeValue(null, "name");
    if (folderName == null)
        throw new XMLStreamException("Missing 'name' attribute on <folder> element", reader.getLocation());
    List<String> folderPath = new ArrayList<>(parentPaths.size() + 1);
    folderPath.addAll(parentPaths);
    folderPath.add(folderName);
    while (reader.nextTag() != XMLStreamConstants.END_ELEMENT) {
        String tagName = reader.getLocalName();
        if (tagName.equals("folder")) {
            parseFolder(reader, folderPath);
        } else if (tagName.equals("book")) {
            parseBook(reader, folderPath);
        } else {
            throw new XMLStreamException("Expected <folder> or <book> element, found: " + tagName, reader.getLocation());
        }
    }
}

private static void parseBook(XMLStreamReader reader, List<String> folderPath) throws XMLStreamException {
    String bookName = reader.getAttributeValue(null, "name");
    if (bookName == null)
        throw new XMLStreamException("Missing 'name' attribute on <book> element", reader.getLocation());
    while (reader.nextTag() != XMLStreamConstants.END_ELEMENT) {
        String tagName = reader.getLocalName();
        if (tagName.equals("chapter")) {
            parseChapter(reader, folderPath, bookName);
        } else {
            throw new XMLStreamException("Expected <chapter> element, found: " + tagName, reader.getLocation());
        }
    }
}

private static void parseChapter(XMLStreamReader reader, List<String> folderPath, String bookName) throws XMLStreamException {
    String chapterName = reader.getAttributeValue(null, "name");
    if (chapterName == null)
        throw new XMLStreamException("Missing 'name' attribute on <chapter> element", reader.getLocation());
    if (! reader.getElementText().isEmpty())
        throw new XMLStreamException("<chapter> element must be empty", reader.getLocation());
    System.out.println("Found:");
    System.out.println("  Folder:  " + folderPath);
    System.out.println("  Book:    " + bookName);
    System.out.println("  Chapter: " + chapterName);
}

Output

Found:
  Folder:  [A, B]
  Book:    Learn Java
  Chapter: Hello, World!
Found:
  Folder:  [A, B]
  Book:    Learn Java
  Chapter: Variables and Types

Commode

I'm having a bit of difficulty still, due to the fact that there are different levels of nesting. It can go Response Root -> folder -> Query Item, or it can go Response Root -> folder -> folder -> folder -> Query Item. How would I get the pointer to move across both possiblities? (In your example, how would I read a book in folder A?) – Ferbam

@Ferbam Add one to the XML. The code already supports it. Note how method parseFolder calls itself recursively, to support unlimited nesting. – Commode