How can i save a file i download using fetch with fs

Question

I try downloading files with the fetch() function from github.
Then i try to save the fetched file Stream as a file with the fs-module.
When doing it, i get this error:

TypeError [ERR_INVALID_ARG_TYPE]: The "transform.writable" property must be an instance of WritableStream. Received an instance of WriteStream

My problem is, that i don't know the difference between WriteStream and WritableStream or how to convert them.

This is the code i run:

async function downloadFile(link, filename = "download") {
    var response = await fetch(link);
    var body = await response.body;
    var filepath = "./" + filename;
    var download_write_stream = fs.createWriteStream(filepath);
    console.log(download_write_stream.writable);
    await body.pipeTo(download_write_stream);
}

Node.js: v18.7.0

Upward

Answer 1

Good question. Web streams are something new, and they are different way of handling streams. WritableStream tells us that we can create WritableStreams as follows:

import {
  WritableStream
} from 'node:stream/web';

const stream = new WritableStream({
  write(chunk) {
    console.log(chunk);
  }
});

Then, you could create a custom stream that writes each chunk to disk. An easy way could be:

const download_write_stream = fs.createWriteStream('./the_path');


const stream = new WritableStream({
  write(chunk) {
    download_write_stream.write(chunk);
  },
});

async function downloadFile(link, filename = 'download') {
  const response = await fetch(link);
  const body = await response.body;
  await body.pipeTo(stream);
}

Underprop

Do you have an explanation for why you can't body.pipeTo(process.stdout)? Your example of creating a WritableStream and calling process.stdout.write inside works, but why is it necessary? In other places stdin.pipe(stdout) works -- what's different about readable.pipeTo()? – Haddad

when you wait for response.body, dont it gathers all the filecontents into memory? – Betake

Answer 2

You can use Readable.fromWeb to convert body, which is a ReadableStream from the web streams API, into a NodeJS Readable stream that can be used with the fs methods.

Note that readable.pipe returns another stream instantly. To wait for it to finish, you can use the promise version of stream.finished to convert it into a Promise, or else you could add listeners for the 'finish' and 'error' events to detect success or failure.

const fs = require('fs');
const { Readable } = require('stream');
const { finished } = require('stream/promises');

async function downloadFile(link, filepath = './download') {
    const response = await fetch(link);
    const body = Readable.fromWeb(response.body);
    const download_write_stream = fs.createWriteStream(filepath);
    await finished(body.pipe(download_write_stream));
}

Deenadeenya