Unit/integration testing Asterisk configuration

Question

Unit and integration testing is usually performed as part of a development process, of course. I'm looking for ways to use this methodology in configuration of an existing system, in this case the Asterisk soft PBX.

In the case of Asterisk, the configuration file is as much a programming language as anything else, complete with loops, jumps, conditionals, etc., and can get quite complex. Changes to the configuration often suffers from the same problems as changes to a complex software product - it can be hard to foresee all the effects without tests in place. It's made worse by the fact that the nature of the system is to communicate with external entities, i.e. make phone calls.

I have a few ideas about testing the system using call files (to create specific calls between extensions) while watching the manager interface for generated events. A test could then watch for an expected result, i.e. dialling *99# should result in the Voicemail application getting called.

The flaws are obvious - it doesn't test the actual result, only what the system thinks is the result, and it probably requires some modification of the system under test. It's also really hard to write these tests robustly enough to only trigger on the expected output, especially if the system is in use (i.e. there are other calls in progress).

Is what I want, a testing system for Asterisk, impossible? If not, do you have any ideas about ways to go about this in a reasonable manner? I'm willing to put a fair amount of development time into this and release the result under a friendly license, but I'm unsure about the best way to approach it.

Viticulture

Answer 1

This is obviously an old question, so there's a good chance that when the original answers were posted here that Asterisk did not support unit / integration testing to the extent that it does today (although the Unit Test Framework API went in on 12/22/09, so that, at least, did exist).

The unit testing framework (David's e-mail from the dev list here) lets you execute unit tests directly within Asterisk. Tests are registered with the framework and can be executed / viewed through the CLI. Since this is all part of Asterisk, the tests are compiled into the executable. You do have to configure Asterisk with the --enable-dev-mode option, and mark the tests for compilation using the menuselect tool (some applications, like app_voicemail, automatically register tests - but they're the minority).

Writing unit tests is fairly straight-forward - and while it (obviously) isn't as fully featured as a commercial unit test framework, it gets the job done and can be enhanced as needed.

That most likely isn't what the majority of Asterisk users are going to want to use - although Asterisk developers are highly encouraged to check it out. Both users and developers are probably interested in integration tests, which the Asterisk Test Suite provides. At its core, the Test Suite is a python script that executes other scripts - be they lua, python, etc. The Test Suite comes with a set of python and lua libraries that help to orchestrate and execute multiple Asterisk instances. Test writers can use third party applications such as SIPp or Asterisk interfaces (AMI, AGI) or a combination thereof to test the hosted Asterisk instance(s).

There are close to 200 tests now in the Test Suite, with more being added on a fairly regular basis. You could obviously write your own tests that exercise your Asterisk configuration and have them managed by the Test Suite - if they're generic enough, you could submit them for inclusion in the Test Suite as well.

Note that the Test Suite can be a bit tricky to set up - Leif wrote a good blog post on setting up the Test Suite here.

Tampere

Correct me if I'm wrong, but the Test Suite is for Asterisk developers to test Asterisk itself, while the question was about testing a system built on top of Asterisk – Harbaugh

It can be used for a variety of purposes. The Asterisk Test Suite orchestrates an instance of Asterisk, using the machine configuration + some overrides. That includes driving Asterisk with mock calls, verifying events, etc. A number of people use it to verify their configuration between upgrades; based on OP's question, that seemed to be a possibility. – Tampere

Answer 2

Well, it depends on what you are testing. There are a lot of ways to handle this sort of thing. My preference is to use Asterisk Call Files bundled with dialplan code. EG: Create a callfile to dial some public number, once it is answered, hop back to the specified dialplan context and perform all of my testing logic (play soundfiles, listen for keypresses, etc.)

I wrote an Asterisk call file library which makes this sort of testing EXTREMELY easy. It has a lot of documentation / examples too, check it out here: http://pycall.org/. That may help you.

Good luck!

Pettifogging

That's not a bad library. Most of the tests I envision are of the type "user calls this short number, verify that a call is placed to this other number on the trunk line" or "user calls this number, verify that the call is answered, a sound file played, and a database entry created". The side effects are (relatively) easy to verify, the user experience not so much. :/ – Viticulture

In that case, you'll want to use the AMI (Asterisk Manager Interface), as you can get live status events back as to whether or not the call was made successfully, etc. Check out the docs here: voip-info.org/wiki/view/Asterisk+manager+API – Pettifogging

Answer 3

You could create a set of specific scenarios and use Asterisk's MixMonitor command to record these calls. This would enable you to establish a set of sound recordings that were normative for your system for these tests, and use an automated sound file comparison tool (Perhaps something from comparing-sound-files-if-not-completely-identical?) to examine the results. Just an idea.

Zig

That won't work. There are a lot of variables when dealing with sound on calls. Static, timing, and other issues will make this nearly impossible. – Pettifogging

I don't have a lot of experience with audio fingerprinting, and have no idea what granularity the comparison would operate with. However, if you're calling from asterisk box to asterisk box on a local network, which is what I imagine you would use for unit testing, I would guess that a lot of the variables you suggest would be closer to constants. That being said, I too, am unsure how effective this type of unit testing would be, just giving suggestions. : - ). – Zig

Answer 4

Unit testing as opposed to integration testing means your code is supposed to be architectured so the logic itself is insulated from external dependencies. You said "the configuration file is as much a programming language as anything else" but that's the thing --- real languages has not just control flow but abstraction capabilities, which allow you to write the logic in a way that can be unit tested. That's why I keep logic outside of asterisk as much as possible.

For integration testing, script linphonec to drive your application, and grep the asterisk console to see what it's doing. You can use docker, and fire up temporary asterisk instances for each test.

Harbaugh