I want to download an encrypted file from my server, decrypt it and save it locally. I want to decrypt the file and write it locally as it is being downloaded rather than waiting for the download to finish, decrypting it and then putting the decrypted file in an anchor tag. The main reason I want to do this is so that with large files the browser does not have to store hundreds of megabytes or several gigabytes in memory.
JavaScript: Writing to download stream
Asked Answered
This is only going to be possible with a combination of service worker + fetch + stream A few browser has worker and fetch but even fewer support fetch with streaming (Blink)
new Response(new ReadableStream({...}))
I have built a streaming file saver lib to communicate with a service worker in other to intercept network request: StreamSaver.js
It's a little bit different from node's stream here is an example
function unencrypt(){
// should return Uint8Array
return new Uint8Array()
}
// We use fetch instead of xhr that has streaming support
fetch(url).then(res => {
// create a writable stream + intercept a network response
const fileStream = streamSaver.createWriteStream('filename.txt')
const writer = fileStream.getWriter()
// stream the response
const reader = res.body.getReader()
const pump = () => reader.read()
.then(({ value, done }) => {
let chunk = unencrypt(value)
// Write one chunk, then get the next one
writer.write(chunk) // returns a promise
// While the write stream can handle the watermark,
// read more data
return writer.ready.then(pump)
)
// Start the reader
pump().then(() =>
console.log('Closed the stream, Done writing')
)
})
There are also two other way you can get streaming response with xhr, but it's not standard and doesn't mather if you use them (responseType = ms-stream || moz-chunked-arrayBuffer) cuz StreamSaver depends on fetch + ReadableStream any ways and can't be used in any other way
Later you will be able to do something like this when WritableStream + Transform streams gets implemented as well
fetch(url).then(res => {
const fileStream = streamSaver.createWriteStream('filename.txt')
res.body
.pipeThrogh(unencrypt)
.pipeTo(fileStream)
.then(done)
})
It's also worth mentioning that the default download manager is commonly associated with background download so ppl sometimes close the tab when they see the download. But this is all happening in the main thread so you need to warn the user when they leave
window.onbeforeunload = function(e) {
if( download_is_done() ) return
var dialogText = 'Download is not finish, leaving the page will abort the download'
e.returnValue = dialogText
return dialogText
}
Is implementation of unencrypt methos missing? –
Singlestick
yes, cuz unencrypt can work diffrent, have a look at webcrypto do implement your own decryption –
Thumbstall
Hi, I am using angular and I tries the same solution, I am not able to get download popup even I have not seen any error on console. –
Singlestick
I could able to download in stream, Its lifesaver, Thanks a lot –
Singlestick
Using the StreamSaver JS on client side to stream the data. The file being download is encrypted and data in base64 format. The stream data returned from fetch is in Uint8Array. Is is there anyway way to change it string format ? The reason I am asking once the data is in Unit8Array format, I am finding it difficult to get the string (bas64 encoded data) from Unit8Array when the data involves the multi byte charaters –
Wendy
HI I am facing another problem, When I download file around 1 GB its downloaded twice but when file is small it download once, At large file it makes three calls in background 1)localhost:8443/api/v1.0/ui/appliance/… 2)jimmywarting.github.io/StreamSaver.js/mitm.html?version=2.0.0 3)jimmywarting.github.io/StreamSaver.js/localhost:8443/965011/… But on small file only two calls made (1) and (3), Any idea why its happening? –
Singlestick
could you make a new issue in my repo or post a new SO question with a example and what goes wrong? –
Thumbstall
Thanks for quick response Jimmy Warting, There is one more issue I am having. My system running on self signed certificate. And I want to serve mitm.html from my server because at client setup outer internet might be unavailable. I tried but facing issue to register service worker. Is it possible to use mitm from local server which is running using self sign certificate ? –
Singlestick
New solution has arrived: showSaveFilePicker/FileSystemWritableFileStream, supported in Chrome and all major derivatives (including Edge and Opera) since the end of 2020, and with a shim (written by the author of the other major answer!) for Firefox and Safari, will allow you to do this directly:
async function streamDownloadDecryptToDisk(url, DECRYPT) {
// create readable stream for ciphertext
let rs_src = fetch(url).then(response => response.body);
// create writable stream for file
let ws_dest = window.showSaveFilePicker().then(handle => handle.createWritable());
// create transform stream for decryption
let ts_dec = new TransformStream({
async transform(chunk, controller) {
controller.enqueue(await DECRYPT(chunk));
}
});
// stream cleartext to file
let rs_clear = rs_src.then(s => s.pipeThrough(ts_dec));
return (await rs_clear).pipeTo(await ws_dest);
}
Depending on performance—if you're trying to compete with MEGA, for instance—you might also consider modifying DECRYPT(chunk) to allow you to use ReadableStreamBYOBReader with it:
…zero-copy reading from an underlying byte source. It is used for efficient copying from underlying sources where the data is delivered as an "anonymous" sequence of bytes, such as files.
Hi James, I am working on something similar and unable to figure out how to read chunk of a specific size. When uploading I make chunks of 5MB using slice and encrypt each of them and send it via multipart upload to S3. However when downing, I am unable to define the chunk size. –
Neisa
And when we are processing chunk of a specific size, would it not create issues as the speed of stream generate from fetch would not match the processing. –
Neisa
@Neisa If the disk and CPU cannot keep up with decrypting and storing the files at full network speed, I believe that the browser will throttle the
fetch stream appropriately, just as it does when downloading files normally. –
Timeserver @Neisa The
reader in this case is nothing more than the ReadableStream produced by Fetch itself. It spits out data in whatever chunk size is efficient for downloading, probably to do with the network, at the browser's discretion. If you need to process the data in 32B or 5MB or whatever chunk sizes, you'll need to package such units up yourself, from the stream Fetch gives you. –
Timeserver Thanks @James. That was useful. I create a wrapper around TransformStream that works in the browser. gist.github.com/samaybhavsar/97d2674536c6f64de8b6d2c43085a347 –
Neisa
© 2022 - 2024 — McMap. All rights reserved.
ReadableStream, see JS Promise - instantly retrieve some data from a function that returns a Promise – FaveolateBloborArrayBuffer, as the data is streaming, following any decryption processes; then usecreateObjectURLordata URIof data for user to download when stream is complete – Faveolatethe browser does not have to store hundreds of megabytes or several gigabytes in memory.– Immersionismnodejs. Here, would want for entire stream to be completed and verified before offering a file for download. What if 100MB of file is ok, though last byte is corrupted? – Faveolatebrowserifyto usenodejsat browser. – Faveolatenodejsis not, generally, intended to be used at browser. Was attempting to link to illustrations of using.pipe(). Process data at server, or evenWorker, then offer download. Again, not certain how a file can be downloaded as a stream. Closest have viewed is a.zipfile being populated as it downloads. What is the size of the file that will be downloaded? – FaveolateWhat is the size of the file that will be downloaded?- it's in the question – ImmersionismrequestFileSystem, then use.toURL()to offer download. Though, admittedly, have not tried to append characters to adata URIset ataelementhrefafter user has clicked anchor – FaveolateBlobandArrayBufferdefined; though, again, have not tried appending bytes to aBloborArrayBufferas the bytes are being downloaded to a local file. – Faveolatedata URIafterclickat<a>element havingdownloadattribute, and the bytes were appended to the saved file. So, this may be possible jsfiddle.net/6xazmmpp – Faveolatei < 100000, tab crashed – Faveolate