webRTC how to tell if there is audio

Question

I am using WebRTC with Asterisk, and getting an error about 5% of the time where there is no audio due to an error with signaling. The simple fix is if there is no audio coming through, then stop the connection and try again. I know this is a bandaid while I fix the real issue. For now though, I will bandaid my code.

To get the audio I am doing the following:

var remoteStream = new MediaStream();
peerConnection.getReceivers().forEach((receiver) => {
  remoteStream.addTrack(receiver.track);
});
callaudio.srcObject = remoteStream;
callaudio.play();

The problem here is that the remote stream always adds a track even when there is no audio coming out of the speakers.

If you inspect chrome://webrtc-internals you can see there is no audio being sent, but there is still a receiver. You can look at the media stream, and see there is indeed an audio track. There is everything supporting the fact that I should be hearing something, but I hear nothing 5% of the time.

My solution is to get the data from the receiver track and check if there is anything coming across there, but I have no Idea how to read that data. I have the web audio API working but it only works if there is some sound currently being played. Sometimes the person on the other end does not talk for 10 seconds. I need a way to read the raw data and see that something is going across that. I just want to know is there ANY data on a MediaStream!

If you do remoteStream.getAudioTracks() you get back an audio track because there is one, there is just no audio going across that track.

Yukikoyukio

Just suggestion : Put this "callaudio.play()" in in callback func : oncanplay , oncanplaythrough or onloadeddata . Also you must initialy set audio : true when you send invite / candidate. Then you can control mute, pause or play even on remote side not just local. – Ibo

What did you end up doing for this? – Cybernetics

I didn't really find anything too good. I ended up fixing the signaling problem. Knowing how the audio is going is HARD, but using getStats() on the peer connection worked pretty well. – Yukikoyukio

Answer 1

In the lastest API, receiver.track is present before a connection is made, even if it goes unused, so you shouldn't infer anything from its presence.

There are at least 5 ways to check when audio reception has been negotiated:

1. Retroactively: Check receiver.track.muted. Remote tracks are born muted, and receive an unmute event if/once data arrives:

   audioReceiver.track.onunmute = () => console.log("Audio data arriving!");
   

2. Proactively: Use pc.ontrack. A track event is fired as a result of negotiation, but only for tracks that will receive data. An event for trackEvent.track.kind == "audio" means there will be audio.
3. Automatic: Use the remote stream provided in trackEvent.streams[0] instead of your own (assuming the other side added one in addTrack). RTCPeerConnection populates this one based on what's negotiated only (no audio track present unless audio is received).
4. Unified-plan: Check transceiver.direction: "sendrecv" or "recvonly" means you're receiving something; "sendonly" or "inactive" means you're not.
5. Beyond negotiation: Use getStats() to inspect .packetsReceived of the "inbound-rtp" stats for .kind == "audio" to see if packets are flowing.

The first four are deterministic checks of what's been negotiated. The fifth is only if everything else checks out, but you're still not receiving audio for some reason.

All these work regardless of whether the audio is silent or not, as you requested (your question is really is about what's been negotiated, not about what's audible).

For more on this, check out my blog with working examples in Chrome and Firefox.

Jabber

Some firewalls send 3-4 packets after that drop all other in stream – Boustrophedon

@arthops I think the question is more about what's been negotiated, and the OP being confused about the presence of unused tracks, but I've added a fifth option that talks about verifying packet arrival. Thanks! – Jabber

Answer 2

Simple hack is:

On first second play back to server 1hz tone.

If server got it on first second, server play back 2hz, if no, play back 1hz.

If client not got 2hz back from server, it restart.

Please note, you should be muted while do that.

Boustrophedon

That might actually be possible. I can make a noise on connection to the bridge. – Yukikoyukio