How to implement 3-way conference call video chat with WebRTC Native Code for Android?

Question

I'm trying to implement 3-way video chat inside an Android app using the WebRTC Native Code package for Android (i.e. not using a WebView). I've written a signalling server using node.js and used the Gottox socket.io java client library inside the client app to connect to the server, exchange SDP packets and establish a 2-way video chat connection.

However now I'm having problems going beyond that to a 3-way call. The AppRTCDemo app that comes with the WebRTC native code package demonstrates 2-way calls only (if a 3rd party attempts to join a room a "room full" message is returned).

According to this answer (which doesn't relate to Android specifically), I'm supposed to do it by creating multiple PeerConnections, so each chat participant will connect to the 2 other participants.

However, when I create more than one PeerConnectionClient (a Java class which wraps a PeerConection, which is implemented on the native side in libjingle_peerconnection_so.so), there is an exception thrown from inside the library resulting from a conflict with both of them trying to access the camera:

E/VideoCapturerAndroid(21170): startCapture failed
E/VideoCapturerAndroid(21170): java.lang.RuntimeException: Fail to connect to camera service
E/VideoCapturerAndroid(21170):  at android.hardware.Camera.native_setup(Native Method)
E/VideoCapturerAndroid(21170):  at android.hardware.Camera.<init>(Camera.java:548)
E/VideoCapturerAndroid(21170):  at android.hardware.Camera.open(Camera.java:389)
E/VideoCapturerAndroid(21170):  at org.webrtc.VideoCapturerAndroid.startCaptureOnCameraThread(VideoCapturerAndroid.java:528)
E/VideoCapturerAndroid(21170):  at org.webrtc.VideoCapturerAndroid.access$11(VideoCapturerAndroid.java:520)
E/VideoCapturerAndroid(21170):  at org.webrtc.VideoCapturerAndroid$6.run(VideoCapturerAndroid.java:514)
E/VideoCapturerAndroid(21170):  at android.os.Handler.handleCallback(Handler.java:733)
E/VideoCapturerAndroid(21170):  at android.os.Handler.dispatchMessage(Handler.java:95)
E/VideoCapturerAndroid(21170):  at android.os.Looper.loop(Looper.java:136)
E/VideoCapturerAndroid(21170):  at org.webrtc.VideoCapturerAndroid$CameraThread.run(VideoCapturerAndroid.java:484)

This happens when initializing the local client even before attempting to establish a connection so it's not related to node.js, socket.io or any of the signalling server stuff.

How do I get multiple PeerConnections to share the camera so that I can send the same video to more than one peer?

One idea I had was to implement some kind of singleton camera class to replace VideoCapturerAndroid that could be shared between multiple connections, but I'm not even sure that would work and I'd like to know if there is a way to do 3-way calls using the API before I start hacking around inside the library.

Is it possible and if so, how?

Update:

I tried sharing a VideoCapturerAndroid object between multiple PeerConnectionClients, creating it for the first connection only and passing it into the initialization function for the subsequent ones, but that resulted in this "Capturer can only be taken once!" exception when creating a second VideoTrack from the VideoCapturer object for the second peer connection:

E/AndroidRuntime(18956): FATAL EXCEPTION: Thread-1397
E/AndroidRuntime(18956): java.lang.RuntimeException: Capturer can only be taken once!
E/AndroidRuntime(18956):    at org.webrtc.VideoCapturer.takeNativeVideoCapturer(VideoCapturer.java:52)
E/AndroidRuntime(18956):    at org.webrtc.PeerConnectionFactory.createVideoSource(PeerConnectionFactory.java:113)
E/AndroidRuntime(18956):    at com.example.rtcapp.PeerConnectionClient.createVideoTrack(PeerConnectionClient.java:720)
E/AndroidRuntime(18956):    at com.example.rtcapp.PeerConnectionClient.createPeerConnectionInternal(PeerConnectionClient.java:482)
E/AndroidRuntime(18956):    at com.example.rtcapp.PeerConnectionClient.access$20(PeerConnectionClient.java:433)
E/AndroidRuntime(18956):    at com.example.rtcapp.PeerConnectionClient$2.run(PeerConnectionClient.java:280)
E/AndroidRuntime(18956):    at android.os.Handler.handleCallback(Handler.java:733)
E/AndroidRuntime(18956):    at android.os.Handler.dispatchMessage(Handler.java:95)
E/AndroidRuntime(18956):    at android.os.Looper.loop(Looper.java:136)
E/AndroidRuntime(18956):    at com.example.rtcapp.LooperExecutor.run(LooperExecutor.java:56)

Attempting to share the VideoTrack object between PeerConnectionClients resulted in this error from the native code:

E/libjingle(19884): Local fingerprint does not match identity.
E/libjingle(19884): P2PTransportChannel::Connect: The ice_ufrag_ and the ice_pwd_ are not set.
E/libjingle(19884): Local fingerprint does not match identity.
E/libjingle(19884): Failed to set local offer sdp: Failed to push down transport description: Local fingerprint does not match identity.

Sharing the MediaStream between PeerConnectionClients results in the app abruptly closing, without any error message appearing in the Logcat.

Unbelieving

May I ask what "3-way video" means? – Howling

@Howling a video conference call with 3 partipants – Unbelieving

@Samgak Hi. Can you share full solution? I had problem to connect several audio. – Zebapda

Answer 1

The problem you are having is that PeerConnectionClient is not a wrapper around PeerConnection it contains a PeerConnection.

I noticed this question wasn't answered so I wanted to see if I could help out a bit. I looked into the source code and PeerConnectionClient is very much hard coded for a single remote peer. You would need to create a collection of PeerConnection objects rather then this line:

private PeerConnection peerConnection;

If you look around a bit more you would notice that it gets a bit more complicated then that.

The mediaStream logic in createPeerConnectionInternal should only be done once and you need to share the stream between your PeerConnection objects like this:

peerConnection.addStream(mediaStream);

You can consult the WebRTC spec or take a look at this stackoverflow question to confirm that the PeerConnection type was designed to only handle one peer. It is also somewhat vaguely implied here.

So you only maintain one mediaStream object:

private MediaStream mediaStream;

So again the main idea is one MediaStream object and as many PeerConnection objects as you have peers you want to connect to. So you will not be using multiple PeerConnectionClient objects, but rather modify the single PeerConnectionClient to encapsulate the multi-client handling. If you do want to go with a design of multiple PeerConnectionClient objects for whatever reason you would just have to abstract the media stream logic (and any support types that should only be created once) out of it.

You will also need to maintain multiple remote video tracks rather then the existing one:

private VideoTrack remoteVideoTrack;

You would obviously only care to render the one local camera and create multiple renderers for the remote connections.

I hope this is enough information to get you back on track.

Molehill

Thanks for your answer. I have already tried sharing a MediaStream object between multiple PeerConnectionClients without success, I will try your suggestion of using a single PeerConnectionClient with multiple PeerConnections – Unbelieving

The exceptions you received are from exactly that. Using multiple PeerConnectionClients. This would try to use the camera twice. I realize you said you tried to refactor that part out. I'm just assuming something must have been left out in the refactor as there is a decent amount of logic that you would need to move. Did you only have one stream? Because this will use the videoCapturer mediaStream.addTrack(createVideoTrack(videoCapturer)); – Molehill

Yes, I only have one stream, I tried sharing it between PeerConnectionClients simply by creating them one after the other and passing the MediaStream created by the first one into the rest (in which case the line of code in your comment is not executed). It's not elegant but I was just trying to figure out which things need to be shared and which are per-connection and get it to the stage where I can initialize the PeerConnectionClients without error (before actually connecting to any peers). – Unbelieving

Gotcha. I realized that shortly after I made my comment. :P It definitely sounds like something was missed along the way. The EGL context and localRender object would need to be unique as well. On an unrelated note the room full message you got would likely be server side. – Molehill

Thanks again for your help, I got it working and I've documented the process in a bit more detail in a self-answer. – Unbelieving

Answer 2

With the help of Matthew Sanders' answer I managed to get it working, so in this answer I'm going to describe in more detail one way of adapting the sample code to support video conference calling:

Most of the changes need to be made in PeerConnectionClient, but also in the class that uses PeerConnectionClient, which is where you communicate with the signalling server and set up the connections.

Inside PeerConnectionClient, the following member variables need to be stored per-connection:

private VideoRenderer.Callbacks remoteRender;
private final PCObserver pcObserver = new PCObserver();
private final SDPObserver sdpObserver = new SDPObserver();
private PeerConnection peerConnection;
private LinkedList<IceCandidate> queuedRemoteCandidates;
private boolean isInitiator;
private SessionDescription localSdp;
private VideoTrack remoteVideoTrack;

In my application I needed a maximum of 3 connections (for a 4-way chat), so I just stored an array of each, but you could put them all inside an object and have an array of objects.

private static final int MAX_CONNECTIONS = 3;
private VideoRenderer.Callbacks[] remoteRenders;
private final PCObserver[] pcObservers = new PCObserver[MAX_CONNECTIONS];
private final SDPObserver[] sdpObservers = new SDPObserver[MAX_CONNECTIONS];
private PeerConnection[] peerConnections = new PeerConnection[MAX_CONNECTIONS];
private LinkedList<IceCandidate>[] queuedRemoteCandidateLists = new LinkedList[MAX_CONNECTIONS];
private boolean[] isConnectionInitiator = new boolean[MAX_CONNECTIONS];
private SessionDescription[] localSdps = new SessionDescription[MAX_CONNECTIONS];
private VideoTrack[] remoteVideoTracks = new VideoTrack[MAX_CONNECTIONS];

I added a connectionId field to the PCObserver and SDPObserver classes, and inside the PeerConnectionClient constructor I allocated the observer objects in the array and set the connectionId field for each observer object to its index in the array. All the methods of PCObserver and SDPObserver that reference the member variables listed above should be changed to index into the appropriate array using the connectionId field.

The PeerConnectionClient callbacks need to be changed:

public static interface PeerConnectionEvents {
    public void onLocalDescription(final SessionDescription sdp, int connectionId);
    public void onIceCandidate(final IceCandidate candidate, int connectionId);
    public void onIceConnected(int connectionId);
    public void onIceDisconnected(int connectionId);
    public void onPeerConnectionClosed(int connectionId);
    public void onPeerConnectionStatsReady(final StatsReport[] reports);
    public void onPeerConnectionError(final String description);
}

And also the following PeerConnectionClient methods:

private void createPeerConnectionInternal(int connectionId)
private void closeConnectionInternal(int connectionId)
private void getStats(int connectionId)
public void createOffer(final int connectionId)
public void createAnswer(final int connectionId)
public void addRemoteIceCandidate(final IceCandidate candidate, final int connectionId)
public void setRemoteDescription(final SessionDescription sdp, final int connectionId)
private void drainCandidates(int connectionId)

As with the methods in the observer classes, all these functions need to be changed to use the connectionId to index into the appropriate array of per-connection objects, instead of referencing the single objects they were previously. Any invocations of callback functions need to also be changed to pass the connectionId back.

I replaced createPeerConnection with a new function called createMultiPeerConnection, which is passed an array of VideoRenderer.Callbacks objects for displaying the remote video stream, instead of a single one. The function calls createMediaConstraintsInternal() once and createPeerConnectionInternal() for each of the PeerConnections, looping from 0 to MAX_CONNECTIONS - 1. The mediaStream object is created only on the first call to createPeerConnectionInternal(), just by wrapping the initialization code in an if(mediaStream == null) check.

One complication I encountered was when when the app shuts down and the PeerConnection instances are closed and the MediaStream disposed of. In the sample code the mediaStream is added to a PeerConnection using addStream(mediaStream), but the corresponding removeStream(mediaStream) function is never called (dispose() is called instead). However this creates problems (a ref count assert in MediaStreamInterface in the native code) when there are more than one PeerConnection sharing a MediaStream object because dispose() finalizes the MediaStream, which should only happen when the last PeerConnection is closed. Calling removeStream() and close() is not enough either, because it doesn't fully shut down the PeerConnection and this leads to an assert crash when disposing the PeerConnectionFactory object. The only fix I could find was to add the following code to the PeerConnection class:

public void freeConnection()
{
    localStreams.clear();
    freePeerConnection(nativePeerConnection);
    freeObserver(nativeObserver);
}

And then calling these functions when finalizing each PeerConnection except the last:

peerConnections[connectionId].removeStream(mediaStream);
peerConnections[connectionId].close();
peerConnections[connectionId].freeConnection();
peerConnections[connectionId] = null;

and shutting down the last one like this:

peerConnections[connectionId].dispose();
peerConnections[connectionId] = null;

After modifying PeerConnectionClient, it's necessary to change the signalling code to set up the connections in the right order, passing in the correct connection index to each of the functions and handling the callbacks appropriately. I did this by maintaining a hash between socket.io socket ids and a connection id. When a new client joins the room, each of the existing members sends an offer to the new client and receives an answer in turn. It's also necessary to initialize multiple VideoRenderer.Callbacks objects, pass them in to the PeerConnectionClient instance, and divide up the screen however you want for a conference call.

Unbelieving

I'm glad I could help, and even happier to see your detailed answer that solved your problem! – Molehill

@Unbelieving Nice job, this. May I ask how you can reach freePeerConnection() and freeObserver()? These functions are private, and PeerConnection doesn't have a public constructor as it is created by PeerConnectionFactory. So I guess I can't extend it and the only way that I see is copying the whole library and modifying it. – Monogram

@OliverHausler yes unfortunately that's what I had to do. I couldn't find a way to close the PeerConnections without crashes happening without modifying the PeerConnection class in the library. – Unbelieving

@Unbelieving thank you for this. I'm now able to have 2 or more peer connections at once. I'm left with a question tho. Suppose A is the client, B and C are the peerConnections. How do I make B and C talk to each other as well? I posted a question here #39580153 Any direction will be greatly appreciated. Thank you! – Cappuccino

@samgak, Can I take a look at the PeerConnectionClient class where you defined the CreateMultiPeerConnection method for my reference? I am using Xamarin and I have ported the AppRTC to Xamarin/.NET and it works just as it should. I am looking to establish multipart video/audio conferencing and came across your post. It's interesting you could do it and I am on to that too, It would help if I can take look and keep it for reference. I am building the app in Xamarin for Android and I have used the Xamarin Java Binding Library to bind the libjingle library and its dependencies. – Mortmain

@RamIyer Here is the PeerConnectionClient class with my modifications: pastebin.com/c0YCHS6g – Unbelieving

@samgak: Thank you so much for the super fast response. I will take a look and report back if I am able to find success. By the way, are you also doing multiparty conference more than 3 connections? What do you think would be the performance overheads if the devices are to manage the peers themselves? I am also thinking of a Selective Forwarding Unit (SFU), such as Jitsi Videobridge or a Multipart Control Unit (MCU) such as Licode or a media server such as Kurento. – Mortmain

@RamIyer I never had to deal with more than 3 connections because it was a gaming application with a maximum of 4 players. I could also assume a wired internet connection (although I did test with mobile devices). Bandwidth per device is linear on the number of connections, so performance is going to get worse the more connections you add, as is the chance that at least one will fail to connect. I had a split screen UI showing all streams at all times, but if you only show one at a time you could transmit audio only on the invisible streams. I don't have any experience with using an SFU. – Unbelieving

@RamIyer also, I used a 3rd party STUN/TURN service, saved me a huge headache as running coturn would only work about 3/4 of the time. – Unbelieving

@samgak, Thank you for your comments. I have made the necessary changes to the PeerConnectionClient. Can you also share the CallActivity class? I am planning to set the max connections to 6 and check the performance. I am intending to bring up the other peers as HUD fragment displays and further extending to display the current speaker as the main display (using a cloud based voice/cognitive services - I am not sure how accurate that would be though, especially when there is a lot of chatter). Just want to take a look at your code for reference. – Mortmain

@samgak. I think you have able to implement it for multiple users. I am having the same problem. I have followed this post but couldn't able to make it work. I keep getting the room is full. Can you please pass the code to help me. :) – Viewy

@samgak, You Should make a blog post for this , it will help lots of developers out there :) – Marge