How to use filesystem's createReadStream with Meteor router(NodeJS)

Question

I need to allow the user of my app to download a file with Meteor. Currently what I do is when the user requests to download a file I enter into a "fileRequests" collection in Mongo a document with the file location and a timestamp of the request and return the ID of the newly created request. When the client gets the new ID it imediately goes to mydomain.com/uploads/:id. I then use something like this to intercept the request before Meteor does:

var connect = Npm.require("connect");
var Fiber = Npm.require("fibers");
var path = Npm.require('path');
var fs = Npm.require("fs");
var mime = Npm.require("mime");

__meteor_bootstrap__.app
    .use(connect.query())
    .use(connect.bodyParser()) //I add this for file-uploading
    .use(function (req, res, next) {
        Fiber(function() {

            if(req.method == "GET") {
                // get the id here, and stream the file using fs.createReadStream();
            }
            next();
        }).run();
    });

I check to make sure the file request was made less than 5 seconds ago, and I immediately delete the request document after I've queried it.

This works, and is secure(enough) I think. No one can make a request without being logged in and 5 seconds is a pretty small window for someone to be able to highjack the created request URL but I just don't feel right with my solution. It feels dirty!

So I attempted to use Meteor-Router to accomplish the same thing. That way I can check if they're logged in correctly without doing the 5 second open to the world trickery.

So here's the code I wrote for that:

    Meteor.Router.add('/uploads/:id', function(id) {

    var path = Npm.require('path');
    var fs = Npm.require("fs");
    var mime = Npm.require("mime");

    var res = this.response;

    var file = FileSystem.findOne({ _id: id });

    if(typeof file !== "undefined") {
        var filename = path.basename(file.filePath);
        var filePath = '/var/MeteorDMS/uploads/' + filename;

        var stat = fs.statSync(filePath);

        res.setHeader('Content-Disposition', 'attachment; filename=' + filename);
        res.setHeader('Content-Type', mime.lookup(filePath));
        res.setHeader('Content-Length', stat.size);

        var filestream = fs.createReadStream(filePath);

        filestream.pipe(res);

        return;
    }
});

This looks great, fits right in with the rest of the code and is easy to read, no hacking involved, BUT! It doesn't work! The browser spins and spins and never quite knows what to do. I have ZERO error messages coming up. I can keep using the app on other tabs. I don't know what it's doing, it never stops "loading". If I restart the server, I get a 0 byte file with all the correct headers, but I don't get the data.

Any help is greatly appreciated!!

EDIT:

After digging around a bit more, I noticed that trying to turn the response object into a JSON object results in a circular structure error.

Now the interesting thing about this is that when I listen to the filestream for the "data" event, and attempt to stringify the response object I don't get that error. But if I attempt to do the same thing in my first solution(listen to "data" and stringify the response) I get the error again.

So using the Meteor-Router solution something is happening to the response object. I also noticed that on the "data" event response.finished is flagged as true.

filestream.on('data', function(data) {
    fs.writeFile('/var/MeteorDMS/afterData', JSON.stringify(res));
});

Sheers

We have the same problem. We found out that there are two problems in one. You need to do return false;, this invokes the middleware's next() here: github.com/tmeasday/meteor-router/blob/master/lib/…, but pipe() does not work, too. We are still investigating. – Brinson

Answer 1

The Meteor router installs a middleware to do the routing. All Connect middleware either MUST call next() (exactly once) to indicate that the response is not yet settled or MUST settle the response by calling res.end() or by piping to the response. It is not allowed to do both.

I studied the source code of the middleware (see below). We see that we can return false to tell the middleware to call next(). This means we declare that this route did not settle the response and we would like to let other middleware do their work.

Or we can return a template name, a text, an array [status, text] or an array [status, headers, text], and the middleware will settle the response on our behalf by calling res.end() using the data we returned.

However, by piping to the response, we already settled the response. The Meteor router should not call next() nor res.end().

We solved the problem by forking the Meteor router and making a small change. We replaced the else in line 87 (after if (output === false)) by:

else if (typeof(output)!="undefined") {

See the commit with sha 8d8fc23d9c in my fork.

This way return; in the route method will tell the router to do nothing. Of course you already settled the response by piping to it.

---

Source code of the middleware as in the commit with sha f910a090ae:

// hook up the serving
__meteor_bootstrap__.app
  .use(connect.query()) // <- XXX: we can probably assume accounts did this
  .use(this._config.requestParser(this._config.bodyParser))
  .use(function(req, res, next) {
    // need to wrap in a fiber in case they do something async
    // (e.g. in the database)
    if(typeof(Fiber)=="undefined") Fiber = Npm.require('fibers');

    Fiber(function() {
      var output = Meteor.Router.match(req, res);

      if (output === false) {
        return next();
      } else {
        // parse out the various type of response we can have

        // array can be
        // [content], [status, content], [status, headers, content]
        if (_.isArray(output)) {
          // copy the array so we aren't actually modifying it!
          output = output.slice(0);

          if (output.length === 3) {
            var headers = output.splice(1, 1)[0];
            _.each(headers, function(value, key) {
              res.setHeader(key, value);
            });
          }

          if (output.length === 2) {
            res.statusCode = output.shift();
          }

          output = output[0];
        }

        if (_.isNumber(output)) {
          res.statusCode = output;
          output = '';
        }

        return res.end(output);
      }
    }).run();
  });

Brinson

NOTE: Meanwhile Meteor 6.5 is out and my changed router does not work anymore. – Brinson