NodeJS parseStream, defining a start and end point for a chunk

Question

Confused by Node's filesystem parsing. Here's my code:

var fs = require('fs'),
    xml2js = require('xml2js');

var parser = new xml2js.Parser();

var stream = fs.createReadStream('xml/bigXML.xml');
stream.setEncoding('utf8');

stream.on('data', function(chunk){ 

    parser.parseString(chunk, function (err, result) {
        console.dir(result);
        console.log('Done');
    });
});


stream.on('end', function(chunk){
    // file have been read over,do something...
    console.log("IT'S OVER")
});

This causes...nothing to happen. No output from XML2JS/the parser at all. When I try to console.log(chunk) it seems that the chunks aren't being output in any sort of meaningful chunks based on anything other than perhaps byte size. The output for one 'chunk' is:

<?xml version="1.0" encoding="UTF-8"?>
    <merchandiser xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="merchandiser.xsd">
    <header><merchantId>1237</merchantId><merchantName>NORDSTROM.com</merchantName><createdOn>12/13/2013 23:50:57</createdOn></header>
    <product product_id="52863929">// product info</product>
    <product product_id="26537849">// product info</product>
    <product product_id="25535647">// product info</product>

This chunk has lots and lots of <product> entries from the XML inside of it. The chunk will end somewhere in the middle of a <product> entry and the next chunk will begin from where this left off.

The main question is How do I get the createReadStream to output chunks starting at <product and ending at </product>?

EDIT: for the purposes of getting the proper output, here's what the XML from the beginning to the end of the first <product> looks like:

<?xml version="1.0" encoding="UTF-8" ?>
<merchandiser xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" xsi:noNamespaceSchemaLocation="merchandiser.xsd">
  <header>
    <merchantId>1237</merchantId>
    <merchantName>NORDSTROM.com</merchantName>
    <createdOn>12/13/2013 23:50:57</createdOn>
  </header>
  <product product_id="52863929" name="Teva 'Psyclone' Print Sandal (Baby, Walker &amp; Toddler) Camo/ Dark Olive 6 M" sku_number="52863929" manufacturer_name="Teva" part_number="1001701">
    <category>
      <primary>Toddler Unisex</primary>
      <secondary>Shoes~~Sandals/Slides</secondary>
    </category>
    <URL>
      <product>http://click.linksynergy.com/link?id=LUyP0GcLCGc&amp;offerid=276223.52863929&amp;type=15&amp;murl=http%3A%2F%2Fshop.nordstrom.com%2FS%2F3297406%3Fcm_cat%3Ddatafeed%26cm_pla%3Dshoes%3Asandals%252fslides%26cm_ite%3Dteva_%2527psyclone%2527_print_sandal_%2528baby%252c_walker_%2526_toddler%2529%3A503158_1%26cm_ven%3DLinkshare</product>
      <productImage>http://content.nordstrom.com/imagegallery/store/product/large/0/_6880020.jpg</productImage>
      <buy></buy>
    </URL>
    <description>
      <short>Rugged construction and stylish good looks define a sporty sandal, with the added convenience and security of hook-and-loop closures across the toe and at the instep.Rugged construction and stylish good looks define a sporty sandal, with the added
        convenience and security of h...</short>
      <long>Rugged construction and stylish good looks define a sporty sandal, with the added convenience and security of hook-and-loop closures across the toe and at the instep.Rugged construction and stylish good looks define a sporty sandal, with the added
        convenience and security of hook-and-loop closures across the toe and at the instep. Color(s): camo/ dark olive, daisy blue. Brand: Teva. Style Name: Teva 'Psyclone' Print Sandal (Baby, Walker &amp; Toddler). Style Number: 503158_1.</long>
    </description>
    <discount currency="USD">
      <amount></amount>
      <type>amount</type>
    </discount>
    <price currency="USD">
      <sale begin_date="" end_date="">24.95</sale>
      <retail>24.95</retail>
    </price>
    <brand>Teva</brand>
    <shipping>
      <cost currency="USD">
        <amount>0.00</amount>
        <currency>USD</currency>
      </cost>
      <information></information>
      <availability>Y</availability>
    </shipping>
    <keywords></keywords>
    <upc>737872649135</upc>
    <m1>503158_1.</m1>
    <pixel>http://ad.linksynergy.com/fs-bin/show?id=LUyP0GcLCGc&amp;bids=276223.52863929&amp;type=15&amp;subid=0</pixel>
    <attributeClass class_id="60">
      <Misc></Misc>
      <Product_Type>Shoes</Product_Type>
      <Size>6 M</Size>
      <Material></Material>
      <Color>CAMO/ DARK OLIVE</Color>
      <Gender>Unisex</Gender>
      <Style></Style>
      <Age></Age>
    </attributeClass>
  </product>

Shelton

Answer 1

You have two possibilities to tackle your issue.

As stated by damphat, XML2JS needs the full XML content before it can parse the data. But you have a file stream, which, well, streams data chunk by chunks. The first solution is to convert this stream of data into a nice big Buffer, and then send it to XML2JS. For this purpose, you can use the stream-to package (npm i stream-to) which will convert the file stream into an array of buffers, which we'll then concatenate into one single buffer using Buffer.concat, like this:

var fs = require('fs')
var streamTo = require('stream-to')
var xml2js = require('xml2js')

var file = fs.createReadStream('input.xml')

streamTo.array(file, function (err, arr) {
    if (err) return console.log(err.message)

    var content = Buffer.concat(arr)
    var parser = new xml2js.Parser()
    parser.parseString(content, function (err, res) {
        if (err) return console.log(err.message)
        console.log(res.merchandiser.product)
    })
})

This works quite well, but since it needs to hold the full file into memory, it won't work if your input files are really big. To handle really big files, you need to use a streaming XML parser, such as sax. However sax doesn't create Javascript objects, but is an EventEmitter, and is a bit harder to use since you have to handle all relevant events to build your object on the fly.

You can use for instance the SaXPath library, which supports a small subset of the XPath syntax. This library emits a match event every time it matches the XPath pattern. Here's an example:

var saxpath = require('saxpath')
var fs = require('fs')
var sax = require('sax')

var saxParser = sax.createStream(true)
var streamer = new saxpath.SaXPath(saxParser, '/merchandiser/product')

streamer.on('match', function(xml) {
    console.log(xml);
});

fs.createReadStream('input.xml').pipe(saxParser)

You then have two options:

1. Since you now have the XML that matches only one product at a time, you can use xml2js to parse a single product at a time
2. SaXPath supports multiple recorders: the default recorder listens to sax events and re-creates the corresponding XML (which is what allowed us to use the first solution), but you can roll out your own recorder, that listens to sax events and creates on the fly javascript objects.

Aprylapse

Good post. The issue is that neither solutions work for me, at all. The file is 1.5gb so the buffer solution won't work. And I need to create javascript objects, so according to your post sax won't work either. So....what can I do here?! I can't find a way to get Node to load up the 1.5gb file, I get buffer errors when I try. – Shelton

Alright, then buffers are out of the question then. I edited my answer to give you a couple of options using SaXPath. To be clear: sax will work. I just requires more work on your side to re-create javascript objects from the sax events. But it'll work – Aprylapse

Perfect! Your edited example got it working just fine thanks so much for the detailed explanation too, there's really no good answers to this question anywhere else on Stack/Google so hopefully it'll be helpful to others. – Shelton

You might be able ot help with the follow-up I've posted here because Sax is refusing to include the CDATA contained within the data: #20674411 – Shelton

Answer 2

xml2js is for full loaded xml.

In your case using sax, it is a stream parser:

// install

npm install sax

// this code is for print all product_id

var fs = require('fs');
var sax = require('sax');

var saxStream = sax.createStream();

saxStream.onopentag = function (node) {
    if(node.name === 'PRODUCT'){
        console.log(node.attributes.PRODUCT_ID);
    }
};

fs.createReadStream('xml/bigXML.xml').pipe(saxStream);

ouput:

52863929
26537849
25535647

Judgeship

This is close - please see edit for an example of what's contained within a <product> node. With XML2JS (just using fs and parseString) I was doing a for loop on the results and converting the object to JSON getting all the info inside. Is that possible here with Sax? The onopentag seems to only give a little detail rather than the contents of the <product> itself. – Shelton