What is software fault injection really used for?

Question

My basic understanding of software fault injection is as follows:

One can't run all possible tests to test a software. So one resorts to make modifications in the code and analyzing the degree of errors that are produced from it.

But how is this useful?

Like say we had:

function foo(a, b) {
   return a/b;
}

and we changed it to

function foo(a, b) {
  return Math.floor(a/b);
}

So what of it?

How was this useful at all?

EDIT

@Leo, Say I wrote a software that finds Fibonacci numbers. I write test that look like this: assert(fib(1) == 1);

assert(fib(0) == 0);

assert(fib(3) == 2);

I claim 100% coverage since all lines of code are executed here.

My client runs these tests and all of them pass. So he thinks, "maybe the tests are themselves wrong. let me introduce some changes in them".

So he changes one of them to assert(fib(1) == 5);

and the test fails. What can he conclude from it?

Sorbose

he won't add the changes in the asserts. he will add changes into fib(). You agree that if he changes fib(), it's expected to fail your tests? – Mencius

@Leo, yes, true. So what can he conclude from the failure? He must have changed some logic and so it failed. – Sorbose

you know, some test factories are paid to deliver you test code. Imagine someone that is leaving the company, does not like his boss and decides to end every test with assertTrue(1==1) :-) it may happen all the time :-) – Mencius

Answer 1

Suppose you've hired a company to deliver to you some software and they promised they unit test their code with 90% coverage, which seems to be a great deal.

So you decide to insert errors in these tests, so you expect to see a much lower coverage of passing tests, but, well, after inserting the errors, you discover that it still 90% coverage :-)

How useful are these tests?

for example, is this test right?

@Test
public void testAdd() {
    int result = 0;
    Claszz c = new Claszz();
    int result2 = c.add(-1, 1);
    assertEquals(result,0);
}

Mencius

I'm a little confused here. What does 90% coverage mean? It is impossible to cover any percentage of all possible tests. – Sorbose

true. coverage usually is used to measure the code coverage, not the universe of possible tests that someone can try – Mencius

when you add errors to your code (or tests, what do you want to know is the "false positives" of your code. Maybe you're writing a test but the test itself says everything is ok due to a bug in the test, not in the code. – Mencius

Sorry didn't see your edit earlier. No that test is not right. So let's say he makes some change in it and it fails. He can't necessarily conclude it was because the test was wrong. Maybe he changed something critical (like assertEquals(result, 1);) and it failed. It is not the test's fault, is it? – Sorbose

Answer 2

Software fault injections are basically used to check the code coverage done by the tests. If a set of test data is run on the code and all of them pass it would definitely not mean that the code is 100% bug free. One reason might be that given the data, some part of the code is not reachable. In this case, the code is modified to see if it still works as expected.

This is primarily used to test the exceptions or error handling which might be encountered rarely.

For example, if there a field which only accepts positive number and there is a requirement that on entering negative values error should be displayed.

In this case the property of the field can be set such that the user is not able to enter - in it. So the condition of negative value is not testable.

Here the code will be modified so that the user can enter -ve value and check the error message.

Stellastellar

Answer 3

Well this is an old question but Google got me here so I will just add my views.

Software fault injection and the related estimates are related to test coverage, but I would say not simply as a percentage of code. It is a measure of how good your tests are at uncovering faults. So you inject 100 faults and your tests find 60 of those. You missed 40% of the faults. You might have had 100% code coverage in your tests and still missed those faults. If you had a set of tests all saying assert(1 == 1), you would have missed 100% of your faults.

Of course, there is then the question of the quality of the injections (i.e., do they trigger observable faults, is the fault important enough to test, is the code reachable, etc). But that is another matter.

Finally, another aspect is robustness testing, with tools like Chaos Monkey that kills VM instances from your live system, aiming to evaluate how your system handles failures, recovery, etc. But this is a slightly different viewpoint since you inject faults at the system level, and I guess this question was on code/unit test level.

Micronucleus