How to chain write stream, immediately with a read stream in Node.js 0.10?

Question

The following line will download an image file from a specified url variable:

var filename = path.join(__dirname, url.replace(/^.*[\\\/]/, ''));
request(url).pipe(fs.createWriteStream(filename));

And these lines will take that image and save to MongoDB GridFS:

 var gfs = Grid(mongoose.connection.db, mongoose.mongo);
 var writestream = gfs.createWriteStream({ filename: filename });
 fs.createReadStream(filename).pipe(writestream);

Chaining pipe like this throws Error: 500 Cannot Pipe. Not Pipeable.

request(url).pipe(fs.createWriteStream(filename)).pipe(writestream);

This happens because the image file is not ready to be read yet, right? What should I do to get around this problem?Error: 500 Cannot Pipe. Not Pipeable.

Using the following: Node.js 0.10.10, mongoose, request and gridfs-stream libraries.

Kantor

Answer 1

request(url).pipe(fs.createWriteStream(filename)).pipe(writestream);

is the same as this:

var fileStream = fs.createWriteStream(filename);
request(url).pipe(fileStream);
fileStream.pipe(writestream);

So the issue is that you are attempting to .pipe one WriteStream into another WriteStream.

Twocycle

Yes, you can only read from readable or duplex streams not writable. See github.com/joyent/node/pull/4843. To chain pipes the stream needs to be duplex i.e. both readable and writable – Valdis

Answer 2

// create 'fs' module variable
var fs = require("fs");

// open the streams
var readerStream = fs.createReadStream('inputfile.txt');
var writerStream = fs.createWriteStream('outputfile.txt');

// pipe the read and write operations
// read input file and write data to output file
readerStream.pipe(writerStream);

Delorsedelos

It was actually the opposite: pipe a readable stream to a writeable stream. – Hoahoactzin

Answer 3

I think the confusion in chaining the pipes is caused by the fact that the pipe method implicitly "makes choices" on it's own on what to return. That is:

readableStream.pipe(writableStream) // Returns writable stream
readableStream.pipe(duplexStream) // Returns readable stream

But the general rule says that "You can only pipe a Writable Stream to a Readable Stream." In other words only Readable Streams have the pipe() method.

Cartload

Then whats going on here? In npm axios documentation it says; response.data.pipe(fs.createWriteStream('path-to-file)) As we know that request is readable stream and response is writable stream, and you said that only Readable streams have pipe() method but here we are pipping writable stream to new writestream directly using pipe no read stream involve. This may suggest that pipe method is available to writable streams also. – Epanaphora

@AmmarBayg I have written this post 2 years ago relating to npm version 0.10. Have you checked it on that version also? – Cartload

Answer 4

You cannot chain the ReadStream to the WriteStream because the latter is not duplex, therefore you would do - for a gzipped archive

request.get(url, {
        gzip: true,
        encoding: null
    })
    .pipe(fs.createWriteStream(tmpPath))
    .on('close', function() {
        console.info("downloaded %s", tmpPath);
        fs.createReadStream(tmpPath)
            .pipe(gunzip)
            .pipe(fs.createWriteStream(destPath))
            .on('close', function() {
                console.info("unarchived %s", destPath);
            })
            .on('error', (error) => {
                console.warn("gzip error:%@", error);
            })
    })
    .on('error', (error) => {
        console.warn("download error:%@", error);
    })

Hoahoactzin