Upload progress indicators for fetch?

Question

I'm struggling to find documentation or examples of implementing an upload progress indicator using fetch.

This is the only reference I've found so far, which states:

Progress events are a high-level feature that won't arrive in fetch for now. You can create your own by looking at the Content-Length header and using a pass-through stream to monitor the bytes received.

This means you can explicitly handle responses without a Content-Length differently. And of course, even if Content-Length is there it can be a lie. With streams, you can handle these lies however you want.

How would I write "a pass-through stream to monitor the bytes" sent? If it makes any sort of difference, I'm trying to do this to power image uploads from the browser to Cloudinary.

NOTE: I am not interested in the Cloudinary JS library, as it depends on jQuery and my app does not. I'm only interested in the stream processing necessary to do this with native JavaScript and Github's fetch polyfill.

---

https://fetch.spec.whatwg.org/#fetch-api

Weizmann

@Magix See Aborting a fetch: The Next Generation #447 – Trunk

@Trunk The link above is, again, for using streams in HTTP responses, not requests. – Conni

Very disappointing to see that 4 years later there is still no solution using fetch API: fetch.spec.whatwg.org/#fetch-api it is currently lacking when it comes to request progression (not response progression) – Cornerwise

Modern browsers, no IE: developer.mozilla.org/en-US/docs/Web/API/ReadableStream – Helga

This link seems to have some concepts implemented of what the OP asks for: dev.to/tqbit/… – Guitar

Answer 1

Streams are starting to land in the web platform (https://jakearchibald.com/2016/streams-ftw/) but it's still early days.

Soon you'll be able to provide a stream as the body of a request, but the open question is whether the consumption of that stream relates to bytes uploaded.

Particular redirects can result in data being retransmitted to the new location, but streams cannot "restart". We can fix this by turning the body into a callback which can be called multiple times, but we need to be sure that exposing the number of redirects isn't a security leak, since it'd be the first time on the platform JS could detect that.

Some are questioning whether it even makes sense to link stream consumption to bytes uploaded.

Long story short: this isn't possible yet, but in future this will be handled either by streams, or some kind of higher-level callback passed into fetch().

Klutz

Too bad. Accepting this for now, but when this becomes a reality, I hope that someone else will post an updated solution! :) – Weizmann

@jaffa-the-cake any news? – Hatchery

Update - showing progress with fetch API using streams - twitter.com/umaar/status/917789464658890753/photo/1 – Oat

@EitanPeer Nice. Will a similar thing work for uploading, e.g. POST? – Antivenin

@EitanPeer But, the question is about progress in upload, not in download – Gasometer

it is 2020 now, why still there is no way to do this :( – Wryneck

It's 2021 now and still nothing? – Pearlypearman

Modern browsers, no IE: developer.mozilla.org/en-US/docs/Web/API/ReadableStream... Google has not yet caught up, this question still shows outdated answers. – Helga

Fetch upload streaming is now part of Chrome (experimental) See chromestatus.com/feature/5274139738767360 which will allow reading bytes transferred. Hopefully other browsers implement this too. – Weddle

It's 2022. Where are we at? – Kalle

@1.21gigawatts - we're back to the future – Marchetti

2023 must be the year – Flofloat

It's now 2024 and back from the future. Any updates? – Kalle

Answer 2

fetch: Chrome only

Browsers are working on supporting a ReadableStream as the fetch body. For Chrome, this has been implemented since v105. For other browsers, it's currently not implemented.

(Note that duplex: "half" is currently required in order to use a stream body with fetch.)

A custom TransformStream can be used to track progress. Here's a working example:

warning: this code does not work in browsers other than Chrome

async function main() {
  const blob = new Blob([new Uint8Array(10 * 1024 * 1024)]); // any Blob, including a File
  const uploadProgress = document.getElementById("upload-progress");
  const downloadProgress = document.getElementById("download-progress");

  const totalBytes = blob.size;
  let bytesUploaded = 0;

  // Use a custom TransformStream to track upload progress
  const progressTrackingStream = new TransformStream({
    transform(chunk, controller) {
      controller.enqueue(chunk);
      bytesUploaded += chunk.byteLength;
      console.log("upload progress:", bytesUploaded / totalBytes);
      uploadProgress.value = bytesUploaded / totalBytes;
    },
    flush(controller) {
      console.log("completed stream");
    },
  });
  const response = await fetch("https://httpbin.org/put", {
    method: "PUT",
    headers: {
      "Content-Type": "application/octet-stream"
    },
    body: blob.stream().pipeThrough(progressTrackingStream),
    duplex: "half",
  });
  
  // After the initial response headers have been received, display download progress for the response body
  let success = true;
  const totalDownloadBytes = response.headers.get("content-length");
  let bytesDownloaded = 0;
  const reader = response.body.getReader();
  while (true) {
    try {
      const { value, done } = await reader.read();
      if (done) {
        break;
      }
      bytesDownloaded += value.length;
      if (totalDownloadBytes != undefined) {
        console.log("download progress:", bytesDownloaded / totalDownloadBytes);
        downloadProgress.value = bytesDownloaded / totalDownloadBytes;
      } else {
        console.log("download progress:", bytesDownloaded, ", unknown total");
      }
    } catch (error) {
      console.error("error:", error);
      success = false;
      break;
    }
  }
  
  console.log("success:", success);
}
main().catch(console.error);

upload: <progress id="upload-progress"></progress><br/>
download: <progress id="download-progress"></progress>

workaround: good ol' XMLHttpRequest

Instead of fetch(), it's possible to use XMLHttpRequest to track upload progress — the xhr.upload object emits a progress event.

async function main() {
  const blob = new Blob([new Uint8Array(10 * 1024 * 1024)]); // any Blob, including a File
  const uploadProgress = document.getElementById("upload-progress");
  const downloadProgress = document.getElementById("download-progress");

  const xhr = new XMLHttpRequest();
  const success = await new Promise((resolve) => {
    xhr.upload.addEventListener("progress", (event) => {
      if (event.lengthComputable) {
        console.log("upload progress:", event.loaded / event.total);
        uploadProgress.value = event.loaded / event.total;
      }
    });
    xhr.addEventListener("progress", (event) => {
      if (event.lengthComputable) {
        console.log("download progress:", event.loaded / event.total);
        downloadProgress.value = event.loaded / event.total;
      }
    });
    xhr.addEventListener("loadend", () => {
      resolve(xhr.readyState === 4 && xhr.status === 200);
    });
    xhr.open("PUT", "https://httpbin.org/put", true);
    xhr.setRequestHeader("Content-Type", "application/octet-stream");
    xhr.send(blob);
  });
  console.log("success:", success);
}
main().catch(console.error);

upload: <progress id="upload-progress"></progress><br/>
download: <progress id="download-progress"></progress>

Disoblige

developer.mozilla.org/en-US/docs/Web/API/ReadableStream – Helga

Possible for request body or response body, or both? – Conni

If you run the XHR example code above you'll see it works for both request and response body progress. These are separate event listeners on XMLHttpRequest. For fetch(), response.body is a stream that can be used to track download progress. – Disoblige

WTF?!? There's a new function fetch() that allows to see true server response (e.g. redirect will be seen as redirect instead of silently hidden and followed) but it cannot follow progress of uploads similar to XHR? Is this result of some kind of committee failure or what happened? – Brier

Hello @jtbandes, any way the readable stream we create can be put in FormData instead of being sent as an octet stream? When I do, body = new FormData(); body.append("file", blob.stream().pipeThrough(progressTrackingStream)); res = await fetch(..., {..., body: body}); the browser converts body into a string! So the server receives the literal text [object ...]. – Scenic

The documentation for FormData's append method does not state that it supports appending ReadableStreams, only strings and Blobs, which explains the behavior you're seeing. You would need a replacement for FormData to support ReadableStreams. For example, with a quick search, I found: github.com/form-data/form-data This makes the FormData itself into a stream, so you should be able to pipe it through the progress tracker. – Disoblige

Answer 3

My solution is to use axios, which supports this pretty well:

axios.request({
    method: "post", 
    url: "/aaa", 
    data: myData, 
    onUploadProgress: (p) => {
      console.log(p); 
      //this.setState({
          //fileprogress: p.loaded / p.total
      //})
    }
}).then (data => {
    //this.setState({
      //fileprogress: 1.0,
    //})
})

I have example for using this in react on github.

Conformance

That was my solution as well. Axios seems to fit the mold really well. – Nonfulfillment

Does axios use fetch or XMLHttpRequest under-the-hood? – Acridine

XMLHttpRequest. If you are using this for react native, beware that XMLHttpRequest seems to be VERY VERY slow to parse large json responses when compared to fetch (about 10 times slower, and it freezes the whole ui thread). – Pentavalent

To get progress in % this.setState({ fileprogress: Math.round( (p.loaded * 100) / p.total ) }) – K2

This does not answer the question, especially because axios doesn't use fetch under the hood, and has no such support. I'm literally authoring it now for them so. – Goodwife

I agree that this is not the solution for the specific question but having in consideration that there is not a solution for the specific question I vote up this answer. – Cornerwise

@DerekHenderson If the real answer is "you can't do x in y" then "do x in z instead" could be useful to many people. – Gnostic

also note that this does not work in nodejs – Leavitt

Hello from 2023, throw an upvote: github.com/axios/axios/pull/5146 – Goodwife

Answer 4

As already explained in the other answers, it is not possible with fetch, but with XHR. Here is my a-little-more-compact XHR solution:

const uploadFiles = (url, files, onProgress) =>
  new Promise((resolve, reject) => {
    const xhr = new XMLHttpRequest();
    xhr.upload.addEventListener('progress', e => onProgress(e.loaded / e.total));
    xhr.addEventListener('load', () => resolve({ status: xhr.status, body: xhr.responseText }));
    xhr.addEventListener('error', () => reject(new Error('File upload failed')));
    xhr.addEventListener('abort', () => reject(new Error('File upload aborted')));
    xhr.open('POST', url, true);
    const formData = new FormData();
    Array.from(files).forEach((file, index) => formData.append(index.toString(), file));
    xhr.send(formData);
  });

Works with one or multiple files.

If you have a file input element like this:

<input type="file" multiple id="fileUpload" />

Call the function like this:

document.getElementById('fileUpload').addEventListener('change', async e => {
  const onProgress = progress => console.log('Progress:', `${Math.round(progress * 100)}%`);
  const response = await uploadFiles('/api/upload', e.currentTarget.files, onProgress);
  if (response.status >= 400) {
    throw new Error(`File upload failed - Status code: ${response.status}`);
  }
  console.log('Response:', response.body);
}

Also works with the e.dataTransfer.files you get from a drop event when building a file drop zone.

Reprovable

it may not be useful when you want to show progress for a both file upload and the response (typical scenario is when uploading a big csv file, and then server does some slow conversation whose progress we want to show as well) – Lanlana

Answer 5

Update: as the accepted answer says it's impossible now. but the below code handled our problem for sometime. I should add that at least we had to switch to using a library that is based on XMLHttpRequest.

const response = await fetch(url);
const total = Number(response.headers.get('content-length'));

const reader = response.body.getReader();
let bytesReceived = 0;
while (true) {
    const result = await reader.read();
    if (result.done) {
        console.log('Fetch complete');
        break;
    }
    bytesReceived += result.value.length;
    console.log('Received', bytesReceived, 'bytes of data so far');
}

thanks to this link: https://jakearchibald.com/2016/streams-ftw/

Giulio

Nice, but does it apply to uploads as well? – Cankerworm

@Cankerworm I tried to find out but I wasn't able to do it. and I like to find a way to do this for upload too. – Giulio

Same same, but so far I wasn't too lucky finding/creating a functioning upload example. – Cankerworm

content-length !== length of body. When http compression is used (common for big downloads), the content-length is the size after the http compression, while the length is the size after the file has been extracted. – Fissile

Your code assumes that the content header length specifies the amount of bytes the fetch is going to download. This is not always true, so your code cannot show progress to the user, as bytesReceived becomes bigger than total – Fissile

Moreover, not even the browser knows the actual content length beforehand. All you're going to get is a post-compression progress indicator. For example, if you're downloading a zip file with unevenly distributed compression ratio (some files are random, some are low entropy) you'll notice that the progress indicator is severely skewed. – Sepulveda

@Fissile totalBytes = response.headers.get('Content-Encoding') !== 'gzip' ? Number(response.headers.get('Content-Length')) : null; – Cladoceran

Answer 6

with fetch: now possible with Chrome >= 105 🎉

How to: https://developer.chrome.com/articles/fetch-streaming-requests/

Currently not supported by other browsers (maybe that will be the case when you read this, please edit my answer accordingly)

Feature detection (source)

const supportsRequestStreams = (() => {
  let duplexAccessed = false;

  const hasContentType = new Request('', {
    body: new ReadableStream(),
    method: 'POST',
    get duplex() {
      duplexAccessed = true;
      return 'half';
    },
  }).headers.has('Content-Type');

  return duplexAccessed && !hasContentType;
})();

HTTP >= 2 required

The fetch will be rejected if the connection is HTTP/1.x.

Applejack

This does answer does not answer the question asked. There is no upload progress being tracked here. – Defiant

@Defiant The answer does not provide a working example, that's right. But the article linked does. Feel free to edit my answer – Applejack

Answer 7

I don't think it's possible. The draft states:

it is currently lacking [in comparison to XHR] when it comes to request progression

---

(old answer):
The first example in the Fetch API chapter gives some insight on how to :

If you want to receive the body data progressively:

function consume(reader) {
  var total = 0
  return new Promise((resolve, reject) => {
    function pump() {
      reader.read().then(({done, value}) => {
        if (done) {
          resolve()
          return
        }
        total += value.byteLength
        log(`received ${value.byteLength} bytes (${total} bytes in total)`)
        pump()
      }).catch(reject)
    }
    pump()
  })
}

fetch("/music/pk/altes-kamuffel.flac")
  .then(res => consume(res.body.getReader()))
  .then(() => log("consumed the entire body without keeping the whole thing in memory!"))
  .catch(e => log("something went wrong: " + e))

Apart from their use of the Promise constructor antipattern, you can see that response.body is a Stream from which you can read byte by byte using a Reader, and you can fire an event or do whatever you like (e.g. log the progress) for every of them.

However, the Streams spec doesn't appear to be quite finished, and I have no idea whether this already works in any fetch implementation.

Katelyn

If I read that example correctly, though, this would be for downloading a file via fetch. I'm interested in progress indicators for uploading a file. – Weizmann

Oops, that quote talks about receiving bytes, which confused me. – Katelyn

@Katelyn Note, Promise constructor is not necessary. Response.body.getReader() returns a Promise. See How to solve Uncaught RangeError when download large size json – Trunk

@Trunk yeah, I've fixed it at the source of the quote already. And no, getReader does not return a promise. No idea what this has to do with the post you linked. – Katelyn

@Katelyn Yes, you are correct .getReader()'s .read() method returns a Promise. That is what was trying to convey. The link is to allude to the premise that if progress can be checked for download, progress can be checked for upload. Put together a pattern which returns expected result, to an appreciable degree; that is progress for fetch() upload. Have not found a way to echo a Blob or File object at jsfiddle, probably missing something simple. Testing at localhost uploads file very rapidly, without mimicking network conditions; though just remembered Network throttling. – Trunk

Answer 8

Since none of the answers solve the problem.

Just for implementation sake, you can detect the upload speed with some small initial chunk of known size and the upload time can be calculated with content-length/upload-speed. You can use this time as estimation.

Gambill

Very clever, nice trick to use while we wait for a realtime solution :) – Disembogue

Too risky for me. Wouldn't want to end up like the windows copy file progress bar – Kulun

Not reliable, complex and will show incorrect values. – Peabody

Answer 9

I wrote a library that supports tracking of upload progress for Fetch API.

Usage:

async function upload() {
  const { trackRequestProgress } = await import(
    "https://unpkg.com/fetch-api-progress/dist/index.js?module"
  );

  const blob = new Blob([new Uint8Array(5 * 1024 * 1024)]);

  const uploadProgress = document.getElementById("upload");

  const request = {
    method: "PUT",
    headers: {
      "Content-Type": "application/octet-stream"
    },
    body: blob
  };

  const trackedRequest = trackRequestProgress(request, (progress) => {
    console.log(
      `Uploaded ${progress.loaded} bytes out of ${
        progress.total ?? "unknown"
      }`
    );
    uploadProgress.value = progress.loaded / progress.total;
  });

  if (Object.is(request, trackedRequest)) {
    console.warn("Upload progress is not supported");
    document.getElementById("not-supported").classList.add("visible");
  }

  await fetch("https://httpbin.org/put", trackedRequest);
}
  
upload();

#not-supported {
  display: none;
  color: red;
}

#not-supported.visible {
  display: block;
}

<!DOCTYPE html>
<html lang="en">
  <head>
    <meta charset="UTF-8" />
    <meta name="viewport" content="width=device-width, initial-scale=1.0" />
    <title>fetch-api-progress</title>
  </head>
  <body>
    <div>
      <label for="upload">Upload:</label>
      <progress id="upload" value="0"></progress>
      
      <div id="not-supported">
        Upload progress is not supported in this browser
      </div>
    </div>
  </body>
</html>

Currently, it supports tracking of upload progress in Chrome and Safari. Firefox, unfortunately, does not support ReadableStream as Request.body.

Erine

does the test request url "teil.one" matter? you test only the construction, no? – Antiserum

@Mist, if you mean the function for supportsRequestStreams, then the domain does not matter. It's just a way to check if the platform supports streams in request.body. – Erine

Answer 10

A possible workaround would be to utilize new Request() constructor then check Request.bodyUsed Boolean attribute

The bodyUsed attribute’s getter must return true if disturbed, and false otherwise.

to determine if stream is distributed

An object implementing the Body mixin is said to be disturbed if body is non-null and its stream is disturbed.

Return the fetch() Promise from within .then() chained to recursive .read() call of a ReadableStream when Request.bodyUsed is equal to true.

Note, the approach does not read the bytes of the Request.body as the bytes are streamed to the endpoint. Also, the upload could complete well before any response is returned in full to the browser.

const [input, progress, label] = [
  document.querySelector("input")
  , document.querySelector("progress")
  , document.querySelector("label")
];

const url = "/path/to/server/";

input.onmousedown = () => {
  label.innerHTML = "";
  progress.value = "0"
};

input.onchange = (event) => {

  const file = event.target.files[0];
  const filename = file.name;
  progress.max = file.size;

  const request = new Request(url, {
    method: "POST",
    body: file,
    cache: "no-store"
  });

  const upload = settings => fetch(settings);

  const uploadProgress = new ReadableStream({
    start(controller) {
        console.log("starting upload, request.bodyUsed:", request.bodyUsed);
        controller.enqueue(request.bodyUsed);
    },
    pull(controller) {
      if (request.bodyUsed) {
        controller.close();
      }
      controller.enqueue(request.bodyUsed);
      console.log("pull, request.bodyUsed:", request.bodyUsed);
    },
    cancel(reason) {
      console.log(reason);
    }
  });

  const [fileUpload, reader] = [
    upload(request)
    .catch(e => {
      reader.cancel();
      throw e
    })
    , uploadProgress.getReader()
  ];

  const processUploadRequest = ({value, done}) => {
    if (value || done) {
      console.log("upload complete, request.bodyUsed:", request.bodyUsed);
      // set `progress.value` to `progress.max` here 
      // if not awaiting server response
      // progress.value = progress.max;
      return reader.closed.then(() => fileUpload);
    }
    console.log("upload progress:", value);
    progress.value = +progress.value + 1;
    return reader.read().then(result => processUploadRequest(result));
  };

  reader.read().then(({value, done}) => processUploadRequest({value,done}))
  .then(response => response.text())
  .then(text => {
    console.log("response:", text);
    progress.value = progress.max;
    input.value = "";
  })
  .catch(err => console.log("upload error:", err));

}

Trunk

This achieves absolutely nothing. It is just a very complex syntax for showing progress/spinner and hiding it when request finishes. – Mislay

Answer 11

I think Fetch does not support upload progress tracking. You can use Axios instead

<script src="https://cdn.jsdelivr.net/npm/axios/dist/axios.min.js"></script>
var formData = new FormData();
var imagefile = document.querySelector('#file');
formData.append("image", imagefile.files[0]);
axios.post('upload_file', formData, { 
 onUploadProgress: (progressEvent) => {
   const { loaded, total } = progressEvent;
   let precentage = Math.floor((loaded * 100) / total);
   console.log(precentage);
 },
 headers: {
      'Content-Type': 'multipart/form-data'
    }
}).then((res) => {console.log("got it");});

Saberhagen

Answer 12

There is currently (2023) an NPM package that upgrades fetch, making it quite straightforward to monitor progress. It's called fetch-progress and is available via npmjs. I've found it quite helpful.

Here's the example given in their docs, which illustrates its simplicity:

fetch(this.props.src)
    .then(
      fetchProgress({
        // implement onProgress method
        onProgress(progress) {
          console.log({ progress });
          // A possible progress report you will get
          // {
          //    total: 3333,
          //    transferred: 3333,
          //    speed: 3333,
          //    eta: 33,
          //    percentage: 33
          //    remaining: 3333,
          // }
        },
      })
    )

Frawley

This is for the download progress, not for uploading. – Interoceptor

Answer 13

I fished around for some time about this and just for everyone who may come across this issue too here is my solution:

const form = document.querySelector('form');
const status = document.querySelector('#status');

// When form get's submitted.
form.addEventListener('submit', async function (event) {
    // cancel default behavior (form submit)
    event.preventDefault();

    // Inform user that the upload has began
    status.innerText = 'Uploading..';

    // Create FormData from form
    const formData = new FormData(form);

    // Open request to origin
    const request = await fetch('https://httpbin.org/post', { method: 'POST', body: formData });

    // Get amount of bytes we're about to transmit
    const bytesToUpload = request.headers.get('content-length');

    // Create a reader from the request body
    const reader = request.body.getReader();

    // Cache how much data we already send
    let bytesUploaded = 0;

    // Get first chunk of the request reader
    let chunk = await reader.read();

    // While we have more chunks to go
    while (!chunk.done) {
        // Increase amount of bytes transmitted.
        bytesUploaded += chunk.value.length;

        // Inform user how far we are
        status.innerText = 'Uploading (' + (bytesUploaded / bytesToUpload * 100).toFixed(2) + ')...';

        // Read next chunk
        chunk = await reader.read();
    }
});

Deach

I don't think this is doing what you think it is. fetch returns a Response, not a Request. Everywhere you refer to as request should actually be named response. You're doing everything else correctly, but what you're receiving is a response body and more of a "download" progress than an "upload" progress. – Fuller

Answer 14

const req = await fetch('./foo.json');
const total = Number(req.headers.get('content-length'));
let loaded = 0;
for await(const {length} of req.body.getReader()) {
  loaded = += length;
  const progress = ((loaded / total) * 100).toFixed(2); // toFixed(2) means two digits after floating point
  console.log(`${progress}%`); // or yourDiv.textContent = `${progress}%`;
}

Sluff

I want to give a credit to Benjamin Gruenbaum for the whole answer. Because I learned it from his lecture. – Sluff

@LeonGilyadov Is the lecture available online anywhere? A link to the source would be nice. – Westward

@MarkAmery Here it is: youtube.com/watch?v=Ja8GKkxahCo (the lecture was given in Hebrew) – Sluff

The question is about uploading, not downloading. – Duff

the problem with fetch progress is when you want to upload (there is no problem with download) – Lenny

Answer 15

Key part is ReadableStream ≪obj_response.body≫.

Sample:

let parse=_/*result*/=>{
  console.log(_)
  //...
  return /*cont?*/_.value?true:false
}

fetch('').
then(_=>( a/*!*/=_.body.getReader(), b/*!*/=z=>a.read().then(parse).then(_=>(_?b:z=>z)()), b() ))

You can test running it on a huge page eg https://html.spec.whatwg.org/ and https://html.spec.whatwg.org/print.pdf . CtrlShiftJ and load the code in.

(Tested on Chrome.)

Montymonument

This answer gets minus points but no one explain why give minus point - so I give +1 – Lenny

It gets a -1 from me because it's not relevant to uploading. – Falcongentle

I believe it got -1 because it seems to be writing the minified version javascript – Abutting