Record multi audio tracks available in a stream with MediaRecorder

Question

I'm building a screen recorder plugin for chrome store. I'm adding microphone's audio track to the media stream that contains (Screen's video track + System audio track). So final stream contain 2 audio tracks one of microphone and other one of system audio.

When I'm passing this stream to MediaRecorder(stream), than in final video I can able listen only single audio which is at 0 index in stream.getAudioTracks(), i.e., MediaRecorder is recording only single audio track.

So how to record a stream containing multi audio tracks using MediaRecorder?

Highbred

Have you been able to solve this issue? I'm having the same problem on a videoconference application. I want to save both audio tracks (local and remote) to the same webm file (this file includes the video of the remote end). Do you know if this is possible? – Constitutionally

When multiple audio source is connected, you need to use AudioContext. – Nutcracker

As of the time of writing this, at least Google Chrome only records one track of each kind (audio/video), no matter how many tracks of each kind a stream contains. This is unfortunate in light of how much we depend on video conversation through our private and work life during the pandemic. I advise people to find some relevant Chromium issue and share their user stories, if they want this to be implemented -- such a high hanging fruit, in light of issues present at any given time in Chrome, won't be picked before after the roof collapses, so to speak. – Ignatia

Answer 1

You can have a look at Muaz Khan's Library for multi stream mixing. Or you can go about it something like this:

const screenStream;
const micStream;
const remoteStream;
// merge audio from remote stream and micStream

const audioCtx = new AudioContext();
const source1 = audioCtx.createMediaStreamSource(micStream);
const source2 = audioCtx.createMediaStreamSource(remoteStream);
const destination = audioCtx.createMediaStreamDestination();

//connect sources to destination 
// you can add gain nodes if you want 
source1.connect(destination);
source2.connect(destination);

const outputStream= new MediaStream();
outputStream.addTrack(screenStream.getVideoTracks()[0]);
outputStream.addTrack(destination.stream.getAudioTracks()[0]);

Zamboanga

It's worth mentioning the linked implementation only allows mixing audio tracks. So recording multiple video tracks with a single media recorder is currently not solved broadly. – Ignatia

To mix video tracks you need to render them on a canvas first (arrange them side by side or any way you want) and then capture the stream from canvas @ArmenMichaeli – Suffumigate

I had some weird issues with Chrome if any of this was executed without an User interaction with the website first. No warning or error. Just the sound was not recorded. – Resonance

Answer 2

Better solution for multiple audio stream in mediaRecord for WebRTC call, which I found :

const startRecording = async () => {
    try {
        const screenStream = await navigator.mediaDevices.getDisplayMedia({
            video: true,
            audio: true,
        });

        const audioCtx = new AudioContext();
        const source1 = audioCtx.createMediaStreamSource(stream.current.srcObject);
        const source2 = audioCtx.createMediaStreamSource(participantStream.current.srcObject);
        const destination = audioCtx.createMediaStreamDestination();
        source1.connect(destination);
        source2.connect(destination);

        const outputStream = new MediaStream();
        outputStream.addTrack(screenStream.getVideoTracks()[0]);
        outputStream.addTrack(destination.stream.getAudioTracks()[0]);
        mediaRecorderRef.current = new MediaRecorder(outputStream);
        mediaRecorderRef.current.ondataavailable = (event) => {
            if (event.data.size > 0) {
                recordedChunksRef.current.push(event.data);
            }
        };
        mediaRecorderRef.current.onstop = () => {
            console.log("dddd");
            setActiveOption({ index: -1 });

            const recordedBlob = new Blob(recordedChunksRef.current, {
                type: "video/mp4",
            });

            const downloadLink = document.createElement("a");
            downloadLink.href = URL.createObjectURL(recordedBlob);
            downloadLink.download = "recorded-screen.mp4";
            document.body.appendChild(downloadLink);
            downloadLink.click();
            document.body.removeChild(downloadLink);

            setRecording(false);
        };

        recordedChunksRef.current = [];
        mediaRecorderRef.current.start();

        screenStream.getVideoTracks()[0].addEventListener("ended", () => {
            setIsOpenRecordingModal(true);
        });

        setRecording(true);
    } catch (error) {
        setActiveOption({ index: -1 });
        console.error("Error accessing screen:", error);
    }
};

Natterjack