how to create a readable stream from a remote url in nodejs?

Question

on nodejs documentation, the streams section says I can do fs.createReadStream(url || path). But, when I actually do that It tells me Error: ENOENT: no such file or directory. I just want to pipe the video from a readable to a writable stream, But I'm stuck on creating a readable one.

my code:

const express = require('express')
const fs = require('fs')
const url = 'https://www.example.com/path/to/mp4Video.mp4'
const port = 3000

app.get('/video', (req, res) => {
    const readable = fs.createReadStream(url)
})
app.listen(port, () => {
    console.log('listening on port ' + port)
})

the ERROR:

listening on port 3000
events.js:291
      throw er; // Unhandled 'error' event
      ^

Error: ENOENT: no such file or directory, open 'https://www.example.com/path/to/mp4Video.mp4'
Emitted 'error' event on ReadStream instance at:
    at internal/fs/streams.js:136:12
    at FSReqCallback.oncomplete (fs.js:156:23) {
  errno: -2,
  code: 'ENOENT',
  syscall: 'open',
  path: 'https://www.example.com/path/to/mp4Video.mp4'
}

PS: https://www.example.com/path/to/mp4Video.mp4 IS NOT THE ACTUAL URL

Latinize

Answer 1

fs.createReadStream() does not work with http URLs only file:// URLs or filename paths. Unfortunately, this is not described in the fs doc, but if you look at the source code for fs.createReadStream() and follow what it calls you can find that it ends up calling fileURULtoPath(url) which will throw if it's not a file: URL.

function fileURLToPath(path) {
  if (typeof path === 'string')
    path = new URL(path);
  else if (!isURLInstance(path))
    throw new ERR_INVALID_ARG_TYPE('path', ['string', 'URL'], path);
  if (path.protocol !== 'file:')
    throw new ERR_INVALID_URL_SCHEME('file');
  return isWindows ? getPathFromURLWin32(path) : getPathFromURLPosix(path);
}

It would suggest using the got() library to get yourself a readstream from a URL:

const got = require('got');
const mp4Url = 'https://www.example.com/path/to/mp4Video.mp4';

app.get('/video', (req, res) => {
    got.stream(mp4Url).pipe(res);
});

More examples described in this article: How to stream file downloads in Nodejs with Got.

---

You can also use the plain http/https modules to get the readstream, but I find got() to be generally useful at a higher level for lots of http request things so that's what I use. But, here's code with the https module.

const https = require('https');
const mp4Url = 'https://www.example.com/path/to/mp4Video.mp4';

app.get("/", (req, res) => {
    https.get(mp4Url, (stream) => {
        stream.pipe(res);
    });
});

More advanced error handling could be added to both cases.

Incarnadine

The require('https') version worked very well without having to install another 3rd party package. – Bercy

Does https.get return the stream? So can I do something like const myStream = https.get(mp4Url)? I'd like to eventually use this with fluent-ffmpeg. – Cornelius

@Cornelius - As you can see in the last code block in my answer, it doesn't return the stream, it provides the stream to the callback you pass it. – Incarnadine

Don't forget to set your content type and also catch finish and error events. – Dyun

How do you go about handling a 302 redirect response? Stream doesn't seem to have a status code – Dyun

@Dyun - Please write your own question, show your own code and describe your own specific circumstance. In this code, a 302 response from the https.get() would just get passed through to the response because https.get() does not automatically follow 302 responses. – Incarnadine

Answer 2

This works for me:

import { get } from 'https'
import { Readable } from 'stream'

export const createUrlReadStream = (url: string): Readable => {
  const readable = new Readable({
    read() {}, // No-op
  })

  get(url, (response) => {
    response.on('data', (chunk: any) => {
      readable.push(chunk)
    })

    response.on('end', () => {
      readable.push(null) // End of stream
    })
  }).on('error', (error) => {
    readable.emit('error', error) // Forward the error to the readable stream
  })

  return readable
}

const url = 'https://www.example.com/path/to/mp4Video.mp4'

const readable = createUrlReadStream(url)

Endamage

Answer 3

A More Modern Solution that works in Node 18+ is to use fetch which returns a readable stream in response.body

import { Readable } from 'stream';
const readableStream = await fetch('https://www.example.com/path/to/mp4Video.mp4').then(r => Readable.fromWeb(r.body));
readableStream.pipe(...)

Loreanloredana

On node 20, this gives the following error Property 'fromWeb' does not exist on type 'typeof Readable'.ts(2339) – Gers

That's strange, it's in the spec: nodejs.org/api/… – Loreanloredana