Extracting and injecting audio to an ongoing VOIP call

Question

I am very new when it comes to VOIP and integrations with VOIP systems.

Here is what I am trying to do:

1. A caller calls in and operator answers the call.
   1.1. Start streaming audio of caller to an analysis service in cloud.
2. Once the audio analysis is performed (generally in a few seconds), operator will press the "Hold" button to perform an action suggested by the analysis.
   2.1. Depending on the result of analysis, play a particular audio file back to the caller to let them know that the operator is doing "x," "y," or "z" while on hold.

Given my non-experience working with VOIP systems, I am looking for any suggestions / pointers to topics, areas, articles, technologies that can point me to the right direction.

Githens

You may find rtpinsertsound ( tools.kali.org/sniffingspoofing/rtpinsertsound ) useful for the injection. See voip-info.org/converting-rtp-to-audio for the capturing piece. – Pixilated

Answer 1

I could give some general point of view. I would be assuming the SIP-based VOIP which is actually pretty omnipresent (IMS, LTE, 3GPP, etc.).

The VOIP has two parts that you might have spotted while searching:

• SIP (the control plane)
• RTP (the data or payload plane = audio)

In general, there are two approaches the one comes from a peer-to-peer world where every change in media flow is communicated to the other party with REFER doing actually call transfer for any purpose. But that is usually not a prefered way of doing things. Here comes the second approach which is kind of hiding whatever changes on the B-party (called party) side. Such thing is used also in IMS (which is behind the modern GSM networks). The trick is that the A-party (caller) actually reaches the B-party proxy. In terms of SIP, it is B2BUA aka back to back user agent. Which as the name suggests it covers all the magic that happens in the called party network.

The magic is then actually hidden behind that B2BUA which actually behaves as an entity in the middle and thus can manipulate both SIP and RTP.

Therefore this entity can actually fork the audio using an MGW (media gateway) towards the "real" B-Party (a human/operator) as well as directing the audio to the ML/AI/Expert System analysis. This process also incorporates an appropriate control plane events like starting the analytic process attach, actual audio forking (RTP) and also triggering the SIP INVITE for final B-party. Whenever the analysis is concluded then out of band messaging to some "rich" client at the SIP Agent (computer/tablet with SoftPhone) or some CRM system attached to the call centre system. Such a message should inform the B-Party about the result of the analysis.

All the magic is hidden either inside the B2BUA or eventually inside SIP application server which is a generic name for various services like call distribution to call centre agents, voice mail, IVR, etc.

The voice analysis is today used at banks for caller verification, mood analysis and many "smart" audio processing.

In that domain, there are some opensource and proprietary SIP systems. They tend to be somehow complex. And moreover, the logic is pretty different compared to request-response systems (like HTTP). The call is a stateful system with "session" (call ~ Call-ID) and everything is bound to that.

Hope that this can help you.

Lerner

Answer 2

Have you considered using an API based VOIP provider like Plivo?

The realtime streaming part of your use case might be difficult but I bet you could find a decent work around. I used to work there as a solutions engineer so I'm pretty familiar with the APIs. Feel free to message me if you have any questions.

Pru