HTML5: Playing live Opus audio frames without browser plug-in

Question

We need to replace our NPAPI browser plugin with a plugin-less solution. We have 3rd party input device that provides us live audio in the form of Opus 'frames'. We transmit those frames to the browser using binary WebSockets; and then, forward the data to our NPAPI plugin for decoding and audio playback. See picture.

What approach should we take to replace the NPAPI plugin with an HTML5-ish solution given these requirements?

1. Minimize end-to-end latency to no more than 3-5s (assumes 200mS round trip network latency).
2. Provide a means to apply audio filters (client/browser side)

Using the html5 audio tag seems to introduce a huge amount of latency as various browsers will require a certain amount of buffering (15-30s of audio) before beginning playback. We understand Opus may or may not be supported on all browsers. If needed (though we'd rather not to reduce bandwidth), we could encapsulate the Opus frames into a Ogg container within the web service before sending the data to the browser. Looking at one of the demos from html5rocks, HTML5 Audio Playground, it appears as though #2 is possible.

If this is a poor place to ask such a design question, please suggest other forums/groups that might be more appropriate.

Thanks for any help or suggests you might offer.

Agustin

Answer 1

I have a similar situation. I have been using WebSockets and Media Source Extensions to play an MP3 feed in Google Chrome with little latency, but the MP3 codec is not supported by some other browsers when used with MSE. It turns out that most browsers, at least Chrome, Firefox, Opera and Edge, can play Opus natively with MSE, provided that it's properly encapsulated in an MP4 or WebM container.

Packing Opus in Ogg is quite simple, I converted some code I found from JavaScript to C#.

Packing Opus in WebM is somewhat more complicated. I wrote this C# code from scratch, based on the WebM/Matroska and EBML specs. When served via HTTP, it plays correctly in Chrome and Firefox, but VLC appears to be unable to stream Opus/WebM over HTTP. At least Chrome requires that the time sequence starts at 0, so packing on the server side was not a good option since that would require modifications to the distribution system.

Finally I ported this to JavaScript so each client could pack the Opus frames in WebM with timestamps correctly starting at 0. This starts the live stream without prebuffering within a second in Chrome and Firefox. Note that I used a 4 byte header on the incoming websocket packets, because the existing distribution system does not preserve packet boundaries (it was built for an MP3 stream). If you use one websocket frame per opus frame and use a fixed number of samples per frame you could remove this header.

Now all that's left is finding a solution for IE11, Safari and some older mobile browsers...

Dipteral

Will award bounties worth 100 reputation if you add working example to your answer. – Anselma

Answer 2

Since you will need to manage all of the buffering and what not yourself, I think the best option is to actually decode the Opus frames in JavaScript. As a bonus, you will get better browser support.

The Aurora.js project has support for this: https://github.com/audiocogs/opus.js It is a bit experimental though at this stage.

If you didn't have to use Opus and WebSocket, you would have many more options available to you. Standard <audio> tag does not take more than a couple seconds of audio buffer to begin playback in most cases. When it does, it's usually because the content type header is wrong and the player software has to figure out what it is, requiring a larger buffer to work with. Or, there is some sort of other synchronization issue.

Agrology

Interesting, I've tried using the audio tag but I always get 15-30s worth of buffering before playback. – Agustin

@Agustin Try the audio tag near the top right of this page: waug.fm Do you have the same problem there? That's a stream hosted on one of my streaming servers, and it starts instantly for me when I click play. – Agrology

but we don't know how much delay is in the feed. the streaming provider could be buffering quite a bit of data. I, too, get 'instant' playback when I have 250K-500K worth of audio available for the browser, but that is 15-30s wroth of audio. – Agustin

I just looked at your site, that is exactly what is happening, the page is loading the stream even if you don't click 'play' (basically queuing it up locally), even when I click play the instant the page is rendered, the browser has already downloaded 500K. – Agustin

@Agustin Yes, in this case there is a 20-second server-side buffer which is flushed to the client as rapidly as possible. I just wanted to rule out the problem of a network issue, but if that site is buffering before play is clicked, then you're right that it isn't a good test. In any case, I have been able to use audio tags within your latency requirements on completely unbuffered streams. I don't have a stream handy that isn't buffered, but I could set one up if you wanted to try it. Basically, if the codec is identified correctly and frames are lined up, it will start fast. – Agrology

Do you happen to have links to other SO articles or elsewhere that discuss the 'buffering vs codec' issue with the <audio> tag? I'll dig up my old <audio> test site/harness (originally it used PCM-to-mp3) and see if I can make it use opus (I assume I'll need to place it in an Ogg container) – Agustin

Answer 3

Why not just decode OPUS and play PCM on client side with JS. All major browser support PCM. Here you can find both: decoder worker - https://github.com/samirkumardas/another-libopus.js and player - https://github.com/samirkumardas/pcm-player . Tested myself.

Anselma