Problems in writing binary data with node.js

Question

I am trying to write the binary body of a request to a file and failing. The file is created on server but I am unable to open it. I am getting 'Fatal error: Not a png' on Ubuntu. Here is how I am making the request:

curl --request POST --data-binary "@abc.png" 192.168.1.38:8080

And here is how I am trying to save it with the file. The first snippet is a middleware for appending all the data together and second one is the request handler:

Middleware:

app.use(function(req, res, next) {
  req.rawBody = '';
  req.setEncoding('utf-8');
  req.on('data', function(chunk) { 
    req.rawBody += chunk;
  });
  req.on('end', function() {
    next();
  });
});

Handler:

exports.save_image = function (req, res) {
  fs.writeFile("./1.png", req.rawBody, function(err) {
    if(err) {
      console.log(err);
    } else {
      console.log("The file was saved!");
    }
  });
  res.writeHead(200);
  res.end('OK\n');
};

Here's some info which might help. In the middleware, if I log the length of rawBody, it looks to be correct. I am really stuck at how to correctly save the file. All I need is a nudge in the right direction.

Gailgaile

Can you clarify "failing"? Does the file exist on the server but have incorrect content? Does the file not exist at all on the server? Do you get an error message? – Negative

Answer 1

Here is a complete express app that works. I hit it with curl --data-binary @photo.jpg localhost:9200 and it works fine.

var app = require("express")();
var fs = require("fs");
app.post("/", function (req, res) {
  var outStream = fs.createWriteStream("/tmp/upload.jpg");
  req.pipe(outStream);
  res.send();
});
app.listen(9200);

I would just pipe the request straight to the filesystem. As to your actual problem, my first guess is req.setEncoding('utf-8'); as utf-8 is for text data not binary data.

Negative

Yes it is working. However, in some cases, I am only getting partial image. This generally happens over a remote connection. – Gailgaile

Well, that's probably not a problem you can fix in your code. If the network doesn't deliver the packets, nothing you can do in your application code is going to magically synthesize them. You'd need to dig in there to discover exactly what is going wrong, but it's probably not a code problem. – Negative

Maybe you should check content-length header against the incoming data, if not matching reject. – Hathcock

@FaridNouriNeshat If I reject a chunk, I don't think I will be able to assemble full image. I am looking at node stream. Peter's answer will probably nail it for me. – Gailgaile

I meant reject the whole request, if see the end event emitted but received content is not complete, so you just assume the request was ended half finished and don't process it further, server the user with a 4xx status code. – Hathcock

Maybe the problem is that you're calling res.send(); too soon. It could be that the client receives the response before it has finished sending the request and stops sending. – Urita

...or, perhaps more accurately, the server closes the connection before the client has finished sending. – Urita

And if that is the problem, then I think you can fix it by replacing res.send(); with outStream.on('close', function() { res.send(); });. – Urita

Answer 2

For fix your code: I'm with @Peter Lyons, that the error is probably the req.setEncoding('utf-8'); line.

I know the following don't ask your question directly, but proposes an alternative to it by using req.files functionality provided by Express.js, which you are using.

if (req.files.photo && req.files.photo.name) { // Get the temporary location of the file. var tmp_path = req.files.photo.path;

// set where the file should actually exists - in this case it is in the "images" directory.
var target_path = './public/profile/' + req.files.photo.name;

// Move the file from the temporary location to the intended location.
fs.rename(tmp_path, target_path, function (error) {
    if (!error) {
        /*
         * Remove old photo from fs.
         * You can remove the following if you want to.
         */
        fs.unlink('./public/profile/' + old_photo, function () {
            if (error) {
                callback_parallel(error);
            }
            else {
                callback_parallel(null);
            }
        });
    }
    else {
        callback_parallel(error);
    }
});

}

Sternlight

This would have been my preference too but I have to work with an external source of data provided to me which is beyond my control. Thanks for answer. – Gailgaile

req.files property will be defined if the client used an http form with enctype="multipart/form_data" and then issued a submit on that form. If the client used a different mechanism, for instance, an AJAX call, there will be no such property set. – Heptagonal