PeerJS/WebRTC connection fails on rapid chunks transmittion

Question

I'm using PeerJS, but thought that this problem can be about WebRTC in general, hope You can help me out:

I'm trying to write a simple peer-to-peer file sharing. I'm using serialisation: "none" for PeerJS connection DataChannel, as I'm sending just pure ArrayBuffers.
Everything is good with files around 10mb but I have problems sending bigger file (30+ mb), for example after sending aroung 10-20 first chunks of 900mb zip file connection between peers start throwing Connection is not open. You should listen for the "open" event before sending messages. (on the Sender side)

My setup:

File dragged to drag&drop, Sender uses FileReader to read it as ArrayBuffer in chunks of 64x1024 bytes (no difference with 16x1024) and as soon as each chunk is read - it's sent via peer.send(ChunkArrayBuffer).

Reciever creates blob from each recieved chunk, after transmission finished creates a complete blob out of those and gives a link to user.

My peer connection settings:

   var con = peer.connect(peerid, {
        label: "file",
        reliable: true,
       serialization: "none"
   })

My sending function:

function sliceandsend(file, sendfunction) {
    var fileSize = file.size;
    var name = file.name;
    var mime = file.type;
    var chunkSize = 64 * 1024; // bytes
    var offset = 0;

 function readchunk() {
    var r = new FileReader();
    var blob = file.slice(offset, chunkSize + offset);
    r.onload = function(evt) {
        if (!evt.target.error) {
            offset += chunkSize;
            console.log("sending: " + (offset / fileSize) * 100 + "%");
            if (offset >= fileSize) {
                con.send(evt.target.result); ///final chunk
                console.log("Done reading file " + name + " " + mime);
                return;
            }
            else {                    
                con.send(evt.target.result);
            }               
        } else {
            console.log("Read error: " + evt.target.error);
            return; 
        }
        readchunk();
       };
        r.readAsArrayBuffer(blob);
    }
    readchunk();
  }

Any ideas what can cause this?

Update: Setting 50ms Timeout between chunk transmittions helped a bit, 900mb file loading reached 6% (instead of 1 - 2% previously) before started throwing errors. Maybe it's some kind of limit of simultaneous operations through datachannel or overflowing some kind of datachannel buffer?
Update1: Here's my PeerJS connection object with DataChannel object inside it:

Hightension

I had the same problem at some point but don't have it anymore. My code is over at github but written in dart. maybe it helps! I added {'ordered': true, 'reliable': true} to createDataChannel maybe it helps? – Ribal

@Ribal sadly this did not helped, 'ordered' and 'reliable' are already true in DataChannel object inside my peerjs conenction object. I will add my conenction object to question now, can you throw yours here, so i can compare two? – Hightension

there is a link to my github code. I am not using peerjs so I cant really help you here :( For me the FileReader takes ~25-50ms to convert the blob to bytearray and it seems that this is enough to make it work for me. – Ribal

@Ribal I meant connection object which is created at runtime, while you connecting to another peer, the one which you are using to send messages. Ofcourse if you have easy access to it right now. Setting localhost to run your code to look at one object is a bit of hussle honestly. – Hightension

The DataChannel looks identical. – Ribal

@Ribal well, that's sad – Hightension

Answer 1

Good News everyone!
It was a buffer overflow of DataChannel problem, thx to this article http://viblast.com/blog/2015/2/25/webrtc-bufferedamount/

bufferedAmount is a property of DataChannel(DC) object which in the latest Chrome version displays amount of data in bytes being currently in buffer, when it exceedes 16MB - DC is silently closed. Therefore anyone who will encounter this problem need to implement buffering mechanism on application level, which will watch for this property and hold back messages if needed. Also, be aware that in versions of Chrome prior to 37 the same property displays quantity(not size) of messages, and more of that it's broken under windows and displays 0, but with v<37 on overflow DC is not closed - only exception thrown, which can also be caught to indicate buffer overflow.

I made an edit in peer.js unminified code for myself, here you can see both methods in one function (for more of the source code you can look at https://github.com/peers/peerjs/blob/master/dist/peer.js#L217)

DataConnection.prototype._trySend = function(msg) {
var self = this;
function buffering() {
    self._buffering = true;
    setTimeout(function() {
        // Try again.
        self._buffering = false;
        self._tryBuffer();
    }, 100);
    return false;
}
if (self._dc.bufferedAmount > 15728640) {
    return buffering(); ///custom buffering if > 15MB is buffered in DC 
} else {
    try {
        this._dc.send(msg);
    } catch (e) {
        return buffering(); ///custom buffering if DC exception caught
    }
    return true;
 }        
}

Also opened an issue on PeerJS GitHub: https://github.com/peers/peerjs/issues/291

Hightension

Answer 2

Have a look at Transfer a file

This page shows how to transfer a file via WebRTC datachannels.

To accomplish this in an interoperable way, the file is split into chunks which are then transferred via the datachannel. The datachannel is reliable and ordered by default which is well-suited to filetransfers.

Although it doesn't use peerjs it can be adapted (to use peerjs) and the code is easy to follow and works without any issues.

Moonseed