Android WebRTC - Mix AudioTracks for Conference

Question

I'm trying to solve a problem with WebRTC Native Android. I've successfully adapted the AppRTCDemo from a 1-1 call to 1-N calls. Currently I have the following scenario:

• A (me) can talk to/listen to B
• A (me) can talk to/listen to C
• B and C can't talk to or listen to each other.

To accomplish that I (A) have 2 PeerConnections respectively with B and C. I understand I need to mix some how the media streams or audio tracks, or in other words:

• Mix (A, B) -> send to C

• Mix (A, C) -> send to B

Any pointers on how to accomplish that? Thank you

Passus

Why do you need to mix A and B and send to C? Is there any reason for B and C not to establish a connection directly? – Mismatch

@Mismatch Meaning, B and C connects directly to each other? If the client (A) is an Android app, and created 2 peer connections to B and another to C...I don't get how it would be possible to connect B to C directly, as you suggested, from the client? – Passus

Are B and C running the same app? If not, what are they? – Mismatch

I use it with a WebRTC-2-Sip bridge. B and C are calls to phone numbers, which could be a voip deskphone or a desktop softphone. – Passus

Answer 1

I believe that you are sending the same localstream to both the parties (party-B and party-C) in multiconnection because if you try to make different local streams for different connections it will throw error as it will try to access mic and camera again which is not possible(i believe).

You must be creating localstream something like this:

stream = sessionFactory.createLocalMediaStream(LOCAL_MEDIA_ID);

For sending the stream of Party-B to Party-C you can do this:

• step1-- create p2p connection between party-A and party-B
• step2-- create p2p connection between party-A and party-C

• step3-- when you add the localstream you call something like this

  peerconnection2.addStream(myStream)
  

In place of myStream (for peer connection with C) you can add the remoteStreamB(remote stream that you are getting from B). This will add the stream that you are getting from B as localstream in second peerconnection that is with C.

For sending remote stream from Party-C to Party-B:

• Suppose you have peerconnection1 with Party-B. To add the remote stream from C to as local stream first you will have to remove the current tracks and then add the new tracks.

• for this you can do something like this

  AudioTrack t1 = mystream.audioTracks.get(0);
  VideoTrack v1 = myStream.videoTracks.get(0);
  if(myStream.removeTrack(t1)){
   if(myStream.removeTrack(v1)){
    t1 = remoteStreamC.audioTracks.get(0);
    v1 = remoteStreamC.videoTracks.get(0);
  
    myStream.addTrack(t1);
    myStream.addTrack(v1);
   }
  }
  

  By this way you will change the content of the localstream that you are sending to B. Now that stream will be having audio and video tracks coming from remotestreamC.

But to make this process error free you will have to use error handling because when someone breaks the connection (either B or C) you will be getting null tracks.... In that case you will have to immediately send the hangup request to the other party as well.

Bonis

I'll be trying to implement your suggestion shortly. I'll let you know the results. Thank you!! – Passus

you are right. I'm only creating the localStream once. And, your first example, B-Party to C-Party works, at the creation of the C-party peerConnection I pass B-party stream as localMedia to C-party. I've tried the 2nd example C-Party to B-Party with no success. B-party already has localStream attached to it. Once I replace the tracks as you suggested I lose audio on B-Party. Do I need call B-PeerConnection.removeStream(localStream) then B-PeerConnection.addStream(modifiedStream)? In other words, how do I "replace" B-party stream? That should trigger new offer too, correct? – Passus

Ok, you answer got me where I needed. As it turns out onRenegotiationNeeded was being triggered but there was no offers being sent when I added or removed the MediaStream from the PeerConnection. So I'm forcing it to "createOffer". Odd since you'd think it would work since the comment "No need to do anything; AppRTC follows a pre-agreed-upon signaling/negotiation protocol" is in the method body. Now, after creating the 3 way call implementing what you suggested, the audio quality on the peer's end gets significantly degraded. Any ideas why? Thanks – Passus

The solution above produces poor audio quality when adding/removing streams. As I understand the ideal would be merge 2 media streams into one media stream and add that to the peer connection. Any pointers on how to solve this? – Passus

we are changing the mediastreams thats why there will be lags ... you will need to handle the situation for this. something like say user at the other end will not be able to know that there is a lag. About quality of audio... I can only say that it depends on the codec used. we need to ensure here that both the peerconnections are using same codecs. – Bonis

No we are not adding two media streams into one ..... we are simply replacing the tracks of one media stream by the tracks of another media stream. You can also directly add the remote stream coming from party c (say RStreamC) to peerconnection between A and B( say pc1) as pc1.addStream(RStreamC). you also need to delete the previous local stream first by this pc1.removeStream(mystream). – Bonis

Sorry to resurrect an old thread, but did you manage to figure out the garbled audio issue? – Elderly