Websocket frame size limitation

Question

I'm sending huge chunks of JSON data through websockets. The JSON may have over 1000 entries. Due to the frame size limitation, the Websocket protocol automatically splits the JSON into frames, which cannot be helped. As we cannot change the frame size of websockets.

The problem:

When I try to evaluate my JSON using JSON.parse it gives me a parsing error which is obvious because the frames are not complete JSON objects. All this happens in the Websocket onmessage event callback. How can I recieve the huge JSON in differnt frames and still be able to parse it? I have tried to concat the frames in onmessage still the error persists.

Side question:

How to concatinate a broken JSON properly?

Sheepskin

Side answer: in order, without leaving out any chunks... – Shell

Any resolution on this? I am facing the same Issue. – Millwright

Add the buffer size params from this list into your web.xml github.com/Atmosphere/atmosphere/blob/master/modules/cpr/src/… – Sheepskin

Answer 1

A single WebSocket frame, per RFC-6455 base framing, has a maximum size limit of 2^63 bytes (9,223,372,036,854,775,807 bytes ~= 9.22 exabytes) (correction by @Sebastian)

However, a WebSocket message, made up of 1 or more frames, has no limit imposed on it from the protocol level.

Each WebSocket implementation will handle message and frame limits differently. Such as setting maximum messages sizes for whole message (usually for memory consumption reasons), or offering streaming options for large messages to better utilize memory.

But in your case, it is likely that your chosen WebSocket implementation has a bug and is improperly splitting up the JSON message into multiple messages, instead of multiple frames. You can use the network inspection tooling in Chrome or an external tool like Wireshark to confirm this behavior.

Saxtuba

It is not splitting the JSON in multiple messages, it is splitting into multiple frames. I know that only because I have inspected using the Chrome tool. How do I handle it? I'm using Atmosphere Framework. – Sheepskin

Have you try to change the websocket's buffer size? Let's have the discussion on the Atmosphere mailing list, but my suspicion is you need to increase the server websocket buffer size. – Humic

9.22 exabytes... guess I'm gonna need a bigger frame buffer. – Sialkot

Answer 2

var wsServer = new websocket.server({
            httpServer: server,
            maxReceivedFrameSize: 131072,
            maxReceivedMessageSize: 10 * 1024 * 1024,
            autoAcceptConnections: false
        });

Change the default maxFrameSize & MessageSize

Unsex

i get this same error. but im running a client. is this applicable to the client as well..not finding much docs on it – Dotty

Answer 3

Since you are dealing with WS, which is low-level, you need to create an application protocol that deals with data that is sent over multiple WS frames. It is up to you to concatenate the data that is in each WS frame (btw, don't concat the frames... concat the data in each frame).

Basically you are reinventing a file transfer protocol.

Globate