node.js stream array of json to response

Question

I have a REST method that should return a JSON array with certain elements that are read from mongodb (using mongoose).

It should be very simple (in the real case the find method has arguments, but that's no the problem):

OutDataModel.find().stream({transform: JSON.stringify}).pipe(res);

The problem with this approach is that I don't get a valid JSON as the result is like this :

{"id":"1","score":11}{"id":"2","score":12}{"id":"3","score":13}

and I was expecting this :

[{"id":"1","score":11},{"id":"2","score":12},{"id":"3","score":13}]

I have not been able to find a solution, but I am pretty sure there will be a simple one.

What I've tried :

Nothing that I'm proud of but here it goes.

1. before streaming write '[' to the response.
2. Instead of JSON.stringify I provided another method that calls JSON.stringify and adds a ',' at the end
3. in the 'end' event of the stream, I write ']' to the response.

Still is not working with this "solution" as I have a trailing comma at the end like this :

 [{"id":"1","score":11},{"id":"2","score":12},{"id":"3","score":13},]

As I said I am pretty sure there should be a clean solution as it is something that should be quite common.

There will be a lot of concurrent invocations to this method, so I don't want to read everything into memory and then write everything to the response. Each of the invocations will not return a lot of records, but all of them together can be huge. The consumer is a java application with spring, using jackson to parse the JSON.

Please let me know how to do it.

ANSWER

I got it working, by creating a Transform stream as was suggested in the accepted answer.

My stream looks like this :

var arraystream = new stream.Transform({objectMode: true});
arraystream._hasWritten = false;


arraystream._transform = function (chunk, encoding, callback) {
    console.log('_transform:' + chunk);
    if (!this._hasWritten) {
        this._hasWritten = true;
        this.push('[' + JSON.stringify(chunk));

    } else {
        this.push(',' + JSON.stringify(chunk));
    }
    callback();
};

arraystream._flush = function (callback) {
    console.log('_flush:');
    this.push(']');
    callback();

};

and the code to use it :

OutDataModel.find().stream().pipe(arraystream).pipe(res);

Thanks.

Mantra

It sounds like you are working around a feature that is working as intended, because the intended consumer (an app ready to consume massive amounts of data) will not wait for the array to be finished (it may never finish for some services). It would be pretty easy to parse the individual chunks of json in this case on-the-fly with a stack. That being said, I'm not sure what the requirements are for your app. – Aforetime

@AlexMa, thanks for the comment. I will clarify it in the question, as really there will be a lot of concurrent invocations to this method, but each of them will not return a huge amount of data. – Mantra

Answer 1

You're on the right track by implementing your own logic. You could also make use of the ArrayFormatter here which is doing something similar: https://gist.github.com/aheckmann/1403797

The transform function gets called on each document individually -- a Mongoose QueryStream emits a single document per 'data' event but QueryStream isn't treating them semantically as part of any larger array data structure; to get an array-formatted JSON, you really do have to do it yourself (as you have surmised).

Markusmarl

I tried with a similar approach of the link you suggested, and finally got it working. I am updating the question with the solution in case anyone needs it – Mantra

Answer 2

I found a very simple and clean solution here: convert mongoose stream to array

That's my simplified version of the code:

Products
    .find({})
    .lean()
    .stream({
        transform: () => {
            let index = 0;
            return (data) => {
                return (!(index++) ? '[' : ',') + JSON.stringify(data);
            };
        }() // invoke
    })
    .on('end', () => {
        res.write(']');
    })
    .pipe(res);

Melaniemelanin