Merging/mixing two audio streams with WebAudio

Question

I have a MediaStreamSource sourced from navigator.getUserMedia, which is connected to a GainNode, which in turn is connected to a ScriptProcessorNode, like so:

[getUserMediaStream] -> [MediaStreamSource] -> [GainNode] -> ScriptProcessorNode -> (destination)

In my application, it's the ScriptProcessorNode that is doing the main work, which is to process and transfer the raw audio stream over a WebSocket to a remote server (my application is basically a web-based audio recorder) for saving. This works fine.

However, now I am trying to introduce a second MediaStreamSource, this time sourced from a WebRTC PeerConnection. The peer connection in itself works fine, and if route the audio into an element, I can hear the audio coming from my peer. However, I want this second MediaStreamSource to also be piped into the ScriptProcessorNode, effectively recording both audio streams.

What I am looking to do is to mix the two audio streams together before they reach the ScriptProcessorNode. I tried connecting the second MediaStreamSource to the same GainNode (and also to the ScriptProcessorNode directly), but this did not work as neither of those nodes accept more than one input node (even though it never returned any errors on trying to connect the superfluous node). I am trying to achieve something like this:

Something like this:

[MediaStreamSource] -> [Intermediary Node(s)?] -> [GainNode] -> [ScriptProcessorNode] -> [Server]
                     /
[MediaStreamSource] /

I then looked into the WebAudio specification and found that the only node that actually does take multiple inputs is the ChannelMergerNode. However, the specification states that the streams are merged into channels based on the order they are connected to the node, so that the first stream connected on the input will become the left channel of the output and the second stream will become the right channel. From this I take that the result would jus end up being one stream on my left ear and the other on my right. I just want both streams to be merged and mixed equally into a single mono channel.

Is this possible at all with the current WebAudio API?

Thank you very much for the help!

Eirik

Boswell

Answer 1

Did you try combining ChannelMergerNode with down mixing ?

http://www.w3.org/TR/webaudio/#UpMix <-- tricky link, it's about downmix and upmix

Petronia

Answer 2

You can create a a MediaStreamDestination:

audioStreamDestination = audioContext.createMediaStreamDestination();

and then connect multiple nodes to it:

node1.connect(audioStreamDestination)
node2.connect(audioStreamDestination)

you can see how it's done in practice with the input from the microphone and the output of an audioworklet here:

https://github.com/petersalomonsen/javascriptmusic/blob/master/wasmaudioworklet/screenrecorder/screenrecorder.js#L15

Calamine

Answer 3

You're correct that the ChannelMergerNode is not suitable for your use case, as it's designed for merging multiple channels into a single multi-channel stream, not for mixing multiple mono streams into a single mono stream.

To mix two audio streams together, you can use a GainNode as a mixer node. Here's an example of how you can achieve this:

>const mixer = context.createGain();
>mediaStreamSource1.connect(mixer);
>mediaStreamSource2.connect(mixer);
>mixer.connect(gainNode);
>gainNode.connect(scriptProcessorNode);

In this setup, both mediaStreamSource1 and mediaStreamSource2 are connected to the mixer node, which is a GainNode with a gain value of 1 (by default). The mixer node will sum the two input signals, effectively mixing them together.

Note that if you want to control the relative levels of the two input streams, you can adjust the gain values of the mediaStreamSource1 and mediaStreamSource2 nodes before connecting them to the mixer node.

Also, keep in mind that this approach assumes that both input streams have the same sample rate and format. If they don't, you may need to use a MediaStreamAudioSourceNode to resample and convert the formats before mixing them.

I hope this helps

Factorage