Login/Register
WebRTC + IOS + Freeswitch : Can't hear audio
Asked Answered
ioswebrtcvoipfreeswitchmod-verto
S

1

9

I'm trying to implement mod_verto on IOS (calling from iPhone to Desktop). I'm using Google's libjingle library for the RTC side, got it up and running using this excellent tutorial.

  • When making a call from my iPhone, I get the call on the desktop browser using the Verto Communicator (downloaded and running on my local machine).
  • On the iPhone side, I can hear the audio from my desktop, but I hear nothing on the desktop side.

  • If I'm making the call using 2 browser windows (with the Verto Communicator), all works fine.

  • Full Disclosure, I'm using ws:// unsecure websocket to connect to FreeSwitch

Here is my JSONRPC log:

Sending login request: 

{"jsonrpc":"2.0","method":"login","id":1,"params":{"login":"1000@MY-IP-ADDRESS","loginParams":{},"userVariables":{},"passwd":"1234","sessid":"53FB0781-B586-4CDA-98C6-558680663B46"}}

Login Response: 

{"jsonrpc":"2.0","id":1,"result":{"message":"logged in","sessid":"53FB0781-B586-4CDA-98C6-558680663B46"}}

verto.invite (including the iPhone sdp): 

{"jsonrpc":"2.0","method":"verto.invite","id":2,"params":{"dialogParams":{"remote_caller_id_number":"1008","useVideo":false,"useMic":"any","useStereo":false,"tag":"webcam","login":"[email protected]","useCamera":"any","videoParams":{"minFrameRate":30,"minWidth":"1280","minHeight":"720"},"destination_number":"1008","screenShare":false,"caller_id_name":"FreeSWITCH User","caller_id_number":"1000","callID":"0CD433FC-A909-4DF2-BC46-0A4A94E9B800","remote_caller_id_name":"Outbound Call","useSpeak":"any"},"sessid":"53FB0781-B586-4CDA-98C6-558680663B46","sdp":"v=0\r\no=- 8564086442942257834 2 IN IP4 127.0.0.1\r\ns=-\r\nt=0 0\r\na=group:BUNDLE audio video\r\na=msid-semantic: WMS\r\nm=audio 58157 UDP\/TLS\/RTP\/SAVPF 111 103 104 9 102 0 8 106 105 13 127 126\r\nc=IN IP4 82.166.93.197\r\na=rtcp:52576 IN IP4 82.166.93.197\r\na=candidate:3168280865 1 udp 2122260223 11.0.0.244 58157 typ host generation 0\r\na=candidate:1260196625 1 udp 2122194687 10.134.172.254 58951 typ host generation 0\r\na=candidate:3168280865 2 udp 2122260222 11.0.0.244 52576 typ host generation 0\r\na=candidate:1260196625 2 udp 2122194686 10.134.172.254 58945 typ host generation 0\r\na=candidate:4066106833 1 tcp 1518280447 11.0.0.244 60562 typ host tcptype passive generation 0\r\na=candidate:94302177 1 tcp 1518214911 10.134.172.254 60563 typ host tcptype passive generation 0\r\na=candidate:4066106833 2 tcp 1518280446 11.0.0.244 60564 typ host tcptype passive generation 0\r\na=candidate:94302177 2 tcp 1518214910 10.134.172.254 60565 typ host tcptype passive generation 0\r\na=candidate:1610196941 1 udp 1686052607 82.166.93.197 58157 typ srflx raddr 11.0.0.244 rport 58157 generation 0\r\na=candidate:1610196941 2 udp 1686052606 82.166.93.197 52576 typ srflx raddr 11.0.0.244 rport 52576 generation 0\r\na=candidate:2274372738 2 udp 1685987070 176.13.15.205 5834 typ srflx raddr 10.134.172.254 rport 58945 generation 0\r\na=candidate:2274372738 1 udp 1685987071 176.13.15.205 5840 typ srflx raddr 10.134.172.254 rport 58951 generation 0\r\na=ice-ufrag:g8lHDtPwH7m5xRex\r\na=ice-pwd:Q6jcBJNTWAyu0JTuIaQAeNI3\r\na=fingerprint:sha-256 0F:A1:68:51:87:3E:B4:C1:0D:33:97:40:78:22:2A:8C:D2:B6:46:23:F5:99:C9:88:5D:34:DB:E2:C5:94:B3:DD\r\na=setup:actpass\r\na=mid:audio\r\na=extmap:1 urn:ietf:params:rtp-hdrext:ssrc-audio-level\r\na=extmap:3 http:\/\/www.webrtc.org\/experiments\/rtp-hdrext\/abs-send-time\r\na=recvonly\r\na=rtcp-mux\r\na=rtpmap:111 opus\/48000\/2\r\na=fmtp:111 minptime=10; useinbandfec=1\r\na=rtpmap:103 ISAC\/16000\r\na=rtpmap:104 ISAC\/32000\r\na=rtpmap:9 G722\/8000\r\na=rtpmap:102 ILBC\/8000\r\na=rtpmap:0 PCMU\/8000\r\na=rtpmap:8 PCMA\/8000\r\na=rtpmap:106 CN\/32000\r\na=rtpmap:105 CN\/16000\r\na=rtpmap:13 CN\/8000\r\na=rtpmap:127 red\/8000\r\na=rtpmap:126 telephone-event\/8000\r\na=maxptime:60\r\nm=video 61966 UDP\/TLS\/RTP\/SAVPF 100 101 116 117 96\r\nc=IN IP4 82.166.93.197\r\na=rtcp:63816 IN IP4 82.166.93.197\r\na=candidate:3168280865 1 udp 2122260223 11.0.0.244 61966 typ host generation 0\r\na=candidate:1260196625 1 udp 2122194687 10.134.172.254 50435 typ host generation 0\r\na=candidate:3168280865 2 udp 2122260222 11.0.0.244 63816 typ host generation 0\r\na=candidate:1260196625 2 udp 2122194686 10.134.172.254 63396 typ host generation 0\r\na=candidate:4066106833 1 tcp 1518280447 11.0.0.244 60566 typ host tcptype passive generation 0\r\na=candidate:94302177 1 tcp 1518214911 10.134.172.254 60567 typ host tcptype passive generation 0\r\na=candidate:4066106833 2 tcp 1518280446 11.0.0.244 60568 typ host tcptype passive generation 0\r\na=candidate:94302177 2 tcp 1518214910 10.134.172.254 60569 typ host tcptype passive generation 0\r\na=candidate:1610196941 1 udp 1686052607 82.166.93.197 61966 typ srflx raddr 11.0.0.244 rport 61966 generation 0\r\na=candidate:1610196941 2 udp 1686052606 82.166.93.197 63816 typ srflx raddr 11.0.0.244 rport 63816 generation 0\r\na=candidate:2274372738 1 udp 1685987071 176.13.15.205 5879 typ srflx raddr 10.134.172.254 rport 50435 generation 0\r\na=candidate:2274372738 2 udp 1685987070 176.13.15.205 5860 typ srflx raddr 10.134.172.254 rport 63396 generation 0\r\na=ice-ufrag:g8lHDtPwH7m5xRex\r\na=ice-pwd:Q6jcBJNTWAyu0JTuIaQAeNI3\r\na=fingerprint:sha-256 0F:A1:68:51:87:3E:B4:C1:0D:33:97:40:78:22:2A:8C:D2:B6:46:23:F5:99:C9:88:5D:34:DB:E2:C5:94:B3:DD\r\na=setup:actpass\r\na=mid:video\r\na=extmap:2 urn:ietf:params:rtp-hdrext:toffset\r\na=extmap:3 http:\/\/www.webrtc.org\/experiments\/rtp-hdrext\/abs-send-time\r\na=extmap:4 urn:3gpp:video-orientation\r\na=recvonly\r\na=rtcp-mux\r\na=rtpmap:100 VP8\/90000\r\na=rtcp-fb:100 ccm fir\r\na=rtcp-fb:100 nack\r\na=rtcp-fb:100 nack pli\r\na=rtcp-fb:100 goog-remb\r\na=rtcp-fb:100 transport-cc\r\na=rtpmap:101 VP9\/90000\r\na=rtcp-fb:101 ccm fir\r\na=rtcp-fb:101 nack\r\na=rtcp-fb:101 nack pli\r\na=rtcp-fb:101 goog-remb\r\na=rtcp-fb:101 transport-cc\r\na=rtpmap:116 red\/90000\r\na=rtpmap:117 ulpfec\/90000\r\na=rtpmap:96 rtx\/90000\r\na=fmtp:96 apt=100\r\n"}}

Call Created response: 

{"jsonrpc":"2.0","id":2,"result":{"message":"CALL CREATED","callID":"0CD433FC-A909-4DF2-BC46-0A4A94E9B800","sessid":"53FB0781-B586-4CDA-98C6-558680663B46"}}

verto.media invoked: 

{"jsonrpc":"2.0","method":"verto.media","id":637,"params":{"sdp":"v=0\no=FreeSWITCH 1457232832 1457232833 IN IP4 159.203.164.7\ns=FreeSWITCH\nc=IN IP4 159.203.164.7\nt=0 0\na=msid-semantic: WMS TcxpBqoS0j04fOIzkIArKYrlV7LCs9Ub\nm=audio 30784 UDP/TLS/RTP/SAVPF 111 126\na=rtpmap:111 opus/48000/2\na=fmtp:111 useinbandfec=1; minptime=10\na=rtpmap:126 telephone-event/8000\na=silenceSupp:off - - - -\na=ptime:20\na=sendonly\na=fingerprint:sha-256 FE:CD:54:3E:2A:D7:DB:00:57:B7:D4:55:A8:EB:79:08:16:BB:B0:EA:43:44:42:9A:90:01:49:37:7B:31:48:F8\na=setup:active\na=rtcp-mux\na=rtcp:30784 IN IP4 159.203.164.7\na=ice-ufrag:qLh1zzclxONPNyQO\na=ice-pwd:G7g4Drkist37beYsP5jfvlqS\na=candidate:9922185636 1 udp 659136 159.203.164.7 30784 typ host generation 0\na=ssrc:1323504502 cname:bhqCyFkpPbjUPSk0\na=ssrc:1323504502 msid:TcxpBqoS0j04fOIzkIArKYrlV7LCs9Ub a0\na=ssrc:1323504502 mslabel:TcxpBqoS0j04fOIzkIArKYrlV7LCs9Ub\na=ssrc:1323504502 label:TcxpBqoS0j04fOIzkIArKYrlV7LCs9Uba0\nm=video 31380 UDP/TLS/RTP/SAVPF 100\na=rtpmap:100 VP8/90000\na=sendonly\na=fingerprint:sha-256 FE:CD:54:3E:2A:D7:DB:00:57:B7:D4:55:A8:EB:79:08:16:BB:B0:EA:43:44:42:9A:90:01:49:37:7B:31:48:F8\na=setup:active\na=rtcp-mux\na=rtcp:31380 IN IP4 159.203.164.7\nb=AS:1024\na=rtcp-fb:100 ccm fir\na=rtcp-fb:100 nack\na=rtcp-fb:100 nack pli\na=ssrc:594893571 cname:bhqCyFkpPbjUPSk0\na=ssrc:594893571 msid:TcxpBqoS0j04fOIzkIArKYrlV7LCs9Ub v0\na=ssrc:594893571 mslabel:TcxpBqoS0j04fOIzkIArKYrlV7LCs9Ub\na=ssrc:594893571 label:TcxpBqoS0j04fOIzkIArKYrlV7LCs9Ubv0\na=ice-ufrag:2KDK4wDMYuAuVdAZ\na=ice-pwd:YTpxObqpLuBEfig7TKHN6bqU\na=candidate:7508673635 1 udp 659136 159.203.164.7 31380 typ host generation 0\n","callID":"0CD433FC-A909-4DF2-BC46-0A4A94E9B800"}}

verto.answer invoked: 

{"jsonrpc":"2.0","method":"verto.answer","id":638,"params":{"callID":"0CD433FC-A909-4DF2-BC46-0A4A94E9B800"}}

Q:What am I missing in order to hear audio on the browser side ?
Any information is appreciated :)

Update, added the freeswitch log

Update 2 IOS : audio stream code

...
let audioTrack = self.factory.audioTrackWithID("Local-Audio")
self.localMediaStream?.addAudioTrack(audioTrack);
self.peerConnection!.addStream(self.localMediaStream)
...

Update 3 - Partial solution While inspecting my code, I found old code that used to add a video track to my local media stream, disabling this section solves the audio problem, but why? what is wrong with that code?

P.S the Promise class was created by a friend, and mimic the JS Promise approach.

func getUserMedia(mediaOptions:Dictionary<String , Any>? = nil) -> Promise<RTCMediaStream>{
    return Promise<RTCMediaStream>(executor: { (resolve, reject) -> () in
        var cameraID:String?
        self.localMediaStream = self.factory.mediaStreamWithLabel("Local-Meida")

        //if video option is enabled (default true)

        //-------------- Disabling this section solves the audio issues --------------
        if(mediaOptions?["video"] as? Bool ?? true){
            for captureDevice in AVCaptureDevice.devicesWithMediaType(AVMediaTypeVideo){
                if (captureDevice.position == mediaOptions?["devicePosition"] as? AVCaptureDevicePosition ?? AVCaptureDevicePosition.Front){
                    cameraID = captureDevice.localizedName
                    break
                }
            }

            if(cameraID == nil){
                reject(NSError(domain: "No cammera detected", code: 0, userInfo: nil))
            }

            let capturer = RTCVideoCapturer.init(deviceName: cameraID)

            let videoSource = self.factory.videoSourceWithCapturer(capturer, constraints: mediaOptions?["constraints"] as? RTCMediaConstraints ?? nil)

            if let localVideoTrack = self.factory.videoTrackWithID("Local-Video", source: videoSource){
                //!!!! THIS IS THE PROBLEMATIC LINE !!!!
                self.localMediaStream?.addVideoTrack(localVideoTrack)
            }else{
                reject(NSError(domain: "No Video track", code: 0, userInfo: nil))
            }
        }
        //-------------- Disabling this section solves the audio issues --------------

        if(mediaOptions?["audio"] as? Bool ?? true){
            let audioTrack = self.factory.audioTrackWithID("Local-Audio")
            self.localMediaStream?.addAudioTrack(audioTrack);
        }
        self.peerConnection!.addStream(self.localMediaStream)

        resolve(self.localMediaStream!)
    })
}

Debug at problematic line enter image description here

Strikebound answered 6/3, 2016 at 11:46 Comment(14)
Why is your audio ID called Local-Audio instead of audio0? – Geanine
is it an issue?, I thought that is is just an ID. check out this code from AppRTC project on gitHub [localStream addAudioTrack:[_factory audioTrackWithID:@"ARDAMSa0"]]; – Strikebound
It can be not sure though. I need to have a look at my project :) But you can try both of these IDs and check if it is working. – Geanine
I did, it is not working :( – Strikebound
Does this project support Mobile-to-Mobile, so you can try iOS-iOS if that problem occurs? – Geanine
Also try to set ARDAMSa0 for audio and ARDAMSv0 for video everywhere that you are using different ones. – Geanine
In order to access the microphone in the iOS, you need permission from the user. Did you get a pop up relating to the the Microphone access and only when it is accepted, the microphone audio is allowed inside the app. – Airsick
Go to the app Settings on iPhone and ensure that the Microphone is enabled – Towering
Microphone permission is enabled, it was automatically requested by the RTCPeerconnection – Strikebound
Currently mobile to mobile is not implemented – Strikebound
Now to understand the whole scenario, you are using this only for audio right? Or you are including video too? – Geanine
That's right only audio – Strikebound
I know it doesn't quite make sense @ShlomiSchwartz but I need to debug some stuff on your code, in order to find it. So can you provide the basic flow that you do. Where do you set your local description? – Geanine
@ShlomiSchwartz Please can you assist in this issue? https://mcmap.net/q/1318803/-freeswitch-mode-verto-webrtc-android-unable-to-make-call-from-android/10413749 – Genevivegenevra
G
5

Update

It is very difficult to understand what is going wrong with the WebRTC implementation unless you have check Media Server, Web Client and iOS Client in our case.

Your case is an Audio Call, so no video streams need to be included on your localStream, but if you take a look closely you see that you are actually adding a videoTrack on your mobile stream:

 if(mediaOptions?["video"] as? Bool ?? true){
        for captureDevice in AVCaptureDevice.devicesWithMediaType(AVMediaTypeVideo){
            if (captureDevice.position == mediaOptions?["devicePosition"] as? AVCaptureDevicePosition ?? AVCaptureDevicePosition.Front){
                cameraID = captureDevice.localizedName
                break
            }
        }

        if(cameraID == nil){
            reject(NSError(domain: "No cammera detected", code: 0, userInfo: nil))
        }

        let capturer = RTCVideoCapturer.init(deviceName: cameraID)

        let videoSource = self.factory.videoSourceWithCapturer(capturer, constraints: mediaOptions?["constraints"] as? RTCMediaConstraints ?? nil)

        if let localVideoTrack = self.factory.videoTrackWithID("Local-Video", source: videoSource){
            //!!!! THIS IS THE PROBLEMATIC LINE !!!!
            self.localMediaStream?.addVideoTrack(localVideoTrack)
        }else{
            reject(NSError(domain: "No Video track", code: 0, userInfo: nil))
        }
    }

So the line which causes the trouble is: self.localMediaStream?.addVideoTrack(localVideoTrack), because you are attaching the video to the localStream.

Opinions

We may have different scenarios of trouble as I mentioned, here I am listing some opinions based on my experience when we built a similar system:

  1. Your MediaServer may not have an implementation which can redirect and handle your calls in a successful state, because there are additional things added when you attach a video (Please see your session description what you actually send there), and it simply refuses to create a call.
  2. Even if your MediaServer handles the scenario, this will include the right implementation of the Client (both Desktop and Mobile) to conform to signaling of its protocols.
  3. You passed all the tests, and you are now adding video and audio, so you are initiating localStream from mobile, and the same need to be created other way. Then you need to handle events when you add streams, remove stream and other stuff via websockets.

Solution in this case

Remove the part that adds localTrack inside localStream, and then even if you have errors, are not caused by creating your localStream, so this step is currently solved.

Original Answer

Here I have a working version of mine, but adapted for your needs as you use only audio.

Creating and setting up the peerConnection (localSide)

// Connecting to the socket
    .........

// Create PeerConnectionFactory
self.peerConnectionFactory = [[RTCPeerConnectionFactory alloc] init];

RTCMediaConstraints *constraints = [self defaultPeerConnectionConstraints];

// Initialize peerConnection based on a list of ICE Servers
self.peerConnection = [self.peerConnectionFactory peerConnectionWithICEServers:[self getICEServers] constraints:constraints delegate:self];

// Create the localStram which contains the audioTrack
RTCMediaStream *localStream = [self createLocalMediaStream];

// Add this stream to the peerConnection
[self.peerConnection addStream:localStream];

// Please be aware here that I am using blocks, as I created a wrapper for easier maintenance, but you can use createOfferWithDelegate: which will go back at your delegation
NSLog(@"Creating peer offer");
RTCManager *strongSelf = self;
[strongSelf.peerConnection createOfferWithCallback:^(RTCSessionDescription *sdp, NSError *error) {
    if (!error) {
        dispatch_async(dispatch_get_main_queue(), ^{
            NSLog(@"Success at creating offer, now setting local description");
            [strongSelf.peerConnection setLocalDescriptionWithCallback:^(NSError *error) {
                if (!error) {
                    dispatch_async(dispatch_get_main_queue(), ^{
                        NSLog(@"Success at setting local description");
                        // On my type of signalization here I am connected, but yours is based on what type of signalization requires
                    });
                }

            } sessionDescription:sdp];
        });
    }
} constraints:[strongSelf defaultPeerConnectionConstraints]];

Helpers

// Now here we create the stream which contains the audio (Please note the ID)
- (RTCMediaStream *)createLocalMediaStream {
    RTCMediaStream *localStream = [self.peerConnectionFactory mediaStreamWithLabel:@"ARDAMS"];
    [localStream addAudioTrack:[self.peerConnectionFactory audioTrackWithID:@"ARDAMSa0"]];

    return localStream;
}

- (RTCMediaConstraints *)defaultPeerConnectionConstraints {
    // DtlsSrtpKeyAgreement is required for Chrome and Firefox to interoperate.
    NSArray *optionalConstraints = @[[[RTCPair alloc] initWithKey:@"DtlsSrtpKeyAgreement" value:@"true"]];

    RTCMediaConstraints *constraints = [[RTCMediaConstraints alloc] initWithMandatoryConstraints:nil optionalConstraints:optionalConstraints];
    return constraints;
}

Please be aware that your problem might be caused also for not calling stuff on the main thread.

Geanine answered 9/3, 2016 at 2:24 Comment(9)
This is Objective-C code but it should not be difficult to translate it, and I am posting it so we can keep track of what you've implemented or not – Geanine
Wow, thanks for that! I'll check it out and see if it moves me forward. Note that since I use mod_verto as the signaling protocol and FreeSwitch is a middleman for media transfer, the cause could be from it. – Strikebound
@ShlomiSchwartz it is good news to find the trouble, as you are currently not using video. And that is about the video section, but not sure if the MediaServer Signalization and the Web Client support the video which can send a different response that the one the server expected. Also please try to debug self.localMediaStream?.addVideoTrack(localVideoTrack) because localVideoTrack must be different from nil. – Geanine
EridB , I've added a screenshot of the debugger – Strikebound
@ShlomiSchwartz try to put the operation on main thread by putting all the code inside Promise<RTCMediaStream>(executor: { (resolve, reject) -> () in // Your code }) in this scope dispatch_async(dispatch_get_main_queue()) { // Your code goes here} as Audio is not a problem for the threads, but the Video yes. – Geanine
Let us continue this discussion in chat. – Geanine
@ShlomiSchwartz Probably you need to handle this on both platforms for the video, because doesn't make sense to process video where you don't do anything with it. But as you are using only audio and it works, I think we are set :) WebRTC is a mess currently on mobile, no documentation, and it's really hard to try to find something without debugging. – Geanine
@ShlomiSchwartz I updated the answer, in order to match your question and notify other users about some use cases :) – Geanine
@Geanine Please can you assist in this issue? https://mcmap.net/q/1318803/-freeswitch-mode-verto-webrtc-android-unable-to-make-call-from-android/10413749 – Genevivegenevra

© 2022 - 2024 — McMap. All rights reserved.