Is there a way to unit test against side effects?
Asked Answered
O

10

14

Any code can provide side effects. Most of the time, side effects can be a sign of bad design and/or need of refactorisation, but when unit testing I find it hard to test against. Consider the following example:

[Test]
public void TrimAll_Removes_All_Spaces()
{
    // Arrange
    var testSubject = "A    string  with     lots   of     space";
    var expectedResult = "Astringwithlotsofspace";

    // Act
    var result = testSubject.TrimAll();

    // Assert
    Assert.AreEqual(expectedResult, result);
}

that tests the following extension:

public static string TrimAll(this string str)
{
    PokeAround();

    return str.Replace(" ", "");
}

The test will pass, but there is no guard agains side effects. The effects of the call to PokeAround will go completely unnoticed.

Given that you don't know what PokeAround is - it could be anything! - how do you write a test that guards against it? Is it at all possible?

Clarification: There have been a couple of comments about the PokeAround as completely unknown being a very unlikely scenario, since we have the source when we write the test. The reason I asked this question, though, was to find a way to guard against side effects added later on. That is, when I write the test, I might have the exension method look like this:

public static string TrimAll(this string str)
{
    return str.Replace(" ", "");
}

The test passes, all is good. Then, a month later when I'm on vacation, a colleague add's the PokeAround call. I want the test I already wrote to fail because he did.

Osmo answered 30/7, 2010 at 12:24 Comment(4)
The question is nonsensical. You cannot unit test code you don't have access to. "In computer programming, unit testing is a method by which individual units of source code are tested to determine if they are fit for use. A unit is the smallest testable part of an application. In procedural programming a unit may be an individual function or procedure. Unit tests are created by programmers or occasionally by white box testers."Bountiful
@WOPR: See my update for a real-world scenario where this is relevant.Osmo
It's still nonsensical. Now you're trying to extend the concept of unit testing to preventing idiocy in other programmers. Your "future colleague" should have unit tested his changes too. You cannot use unit testing to test for future changes... he/she could have just as well deleted all your code and changed it to delete the master boot record.Bountiful
I suppose there is a way though - set file permission privileges on the code files to stop other coders messing with it.Bountiful
G
14

This is what is called sensing in Working Effectively With Legacy Code. That is, sensing the effects of calling the tested method.

Given that you don't know what PokeAround is - it could be anything!

Since we are talking about unit tests, this should hardly be true - unit testing is whitebox testing, and the code is (should be) there for you to check. Unless it is in some closed source 3rd party library, in which case you don't need to test it can't unit test it by definition (maybe you need functional/acceptance tests, but that is an entirely different matter...).

Update: so you want to make sure that future changes to your unit tested method will never have any unanticipated side effects? I think you

  1. can't,
  2. shouldn't.

You can't, because there is no sensible way to detect the lack of side effects from a method call in a real life (nontrivial) program. What you are looking for is some check that the state of the whole universe has not changed apart from this and this little thing. Even from the point of view of a humble program, that universe is vast. A method call can create/update/delete any number of local objects (many of which you can't even see from your unit test environment), touch files on available local/network file systems, execute DB requests, make remote procedure calls...

You shouldn't, because it is up to your colleague making that future change to take care of unit testing his/her change. If you don't trust that this is going to happen, you have a people or process problem, not a unit testing problem.

Galleass answered 30/7, 2010 at 12:32 Comment(5)
@Péter Török, actually you do need to test 3rd party libraries. The library could, and likely will change at some point. By having tests written that confirm your assumptions about how it works, you can safely upgrade the library, and know if there were any changes in the expected behavior immediately, and not after it goes to production when users start calling.Linguistic
I think the last sentence should be rephrased to: "Unless it is in some 3rd party library, in which case you shouldn't need to test it :-)" (in a perfect world).Ideally
See my update for a real-world scenario where this is relevant.Osmo
@Chad, @strager, fair enough, 3rd party libraries may need to be tested. But that is entirely out of the scope of this concrete question, and is not unit testing anyway. I rephrased my sentence, hopefully making it clearer :-)Lxx
Of course, if there was a way to mark a function as pure (and to flag when it becomes impure) in the language, the answer would be much simpler. =]Ideally
B
4

Keep in mind that unit tests are only one tool in an arsenal of verifications and checks, some of which may catch your colleague's "PokeAround":

  1. Unit tests
  2. Code Inspections/Reviews
  3. Design by Contract
  4. Assertions
  5. Static analysis tools like FindBugs and the Checker Framework (@NonNull, @Pure, and @ReadOnly all rock!)

others?

The test passes, all is good. Then, a month later when I'm on vacation, a colleague add's the PokeAround call. I want the test I already wrote to fail because he did.

What makes you think that your colleague wouldn't change the test as well?

Billie answered 10/12, 2010 at 3:8 Comment(1)
Assumedly the colleague is incompetent, not malicious.Showy
L
2

Given that you don't know what PokeAround is - it could be anything! - how do you write a test that guards against it? Is it at all possible?

This question is specious. The situation is unlikely to occur in the real world.

  1. You always know what PokeAround is. It's unit testing. You have the source.

    If -- through some organizational evil -- you are prohibited from reading the source, you have an organizational problem, not a technical problem.

    If you don't know what PokeAround is, you have people who are being specifically evil and preventing success. They need new jobs. Or you do.

  2. You must use Mocks for this PokeAround so you can observe the side-effects.

"guard against side effects added later on."

This is not an example of a mysterious piece of code. You still know what PokeAround is. You always know what PokeAround is.

This is why we do Regression Testing. http://en.wikipedia.org/wiki/Regression_testing

It's still unit testing.

  • You still test PokeAround with a stand-alone unit test.

  • And you test things that use PokeAround with a mock of PokeAround.

Leonilaleonine answered 30/7, 2010 at 12:41 Comment(5)
"You always know what PokeAround is. It's unit testing. You have the source." -- If I always knew what PokeAround (and any other function I code) was doing then I would have no reason to write Unit-Test at all. I only know what they are supposed to do.Fitz
@Luther Blissett: "If I always knew what PokeAround (and any other function I code) was doing then I would have no reason to write Unit-Test" What? Testing is not discovery. Testing is evidence that it actually does what it claims. Testing is not something you do to discover inner mysteries of code.Leonilaleonine
See my update for a real-world scenario where this is relevant.Osmo
If I know that 'X' does 'Y' then I don't need to test this because it means I have a proof (code properties can be statically prooved, I don't need to write a unit-Test to verify a division routine, if the routine has a formal proof from a theorem solver). If the routine passes its tests, it only means that I don't know if there are arguments to 'X' where 'X' does not 'Y'.Fitz
@Luther Blissett: I don't know why you're talking about built-in methods (like division) that we don't unit test unless we wrote the division routine. Why are you mentioning this? Many folks use unit testing to confirm their static proof.Leonilaleonine
F
2

No first-hand experience from me, but this might interest you: Code Contracts

http://research.microsoft.com/en-us/projects/contracts/

has a provision to put an [pure] attribute to a method and enforce side-effect freeness through runtime or compile time checks. It also allows to specify and enforce an impressing set of other contraints.

Fitz answered 30/7, 2010 at 12:50 Comment(0)
S
1

I don't program in this language, but the following would be one way to test whether the original string was modified:

[Test]
public void TrimAll_Removes_All_Spaces()
{
    // Arrange
    var testSubject = "A    string  with     lots   of     space";
    var testSubjectCopy = "A    string  with     lots   of     space";
    var expectedResult = "Astringwithlotsofspace";

    // Act
    var result = testSubject.TrimAll();

    // Assert
    Assert.AreEqual(expectedResult, result);

    // Check for side effects
    Assert.AreEqual(testSubjectCopy, testSubject);
}
Showy answered 10/12, 2010 at 2:48 Comment(0)
L
0

You could look at it another way, that by writing a unit test for this code you are forcing yourself to acknowledge a problem in the code (side effects). You could then re-write / restructure the code in such a way that it is testable, possibly by moving PokeAround into it's own function, or just relocating it from the code you are trying to test, if it needs more input/state to be testable.

Labellum answered 30/7, 2010 at 12:33 Comment(0)
B
0

I'm not sure you could. After all, whether it is an intended effect or a side effect depends on the designers intention.

I'd suggest you assert that "side-effected" code is unchanged.

Also, a tool like Jester might help, but I don't think it would make a difference in your example.

Blunderbuss answered 30/7, 2010 at 12:34 Comment(0)
S
0

Depends what you mean by side effect - I mean, maybe PokeAround() does something important which needs to be done. How do you classify a side effect?

Anyway, there's no particular technology/technique I'm aware of to guard against side effects as such in a single unit test, but as long as you have test coverage for all of your code, any unwanted side effects will hopefully get picked up by at least one test.

BDD/Integration test tools will also help against this, as they (usually) test larger areas of functionality and not just individual classes/methods.

One thing you might want to look at is Design By Contract (DBC). This lets you specify pre and post conditions, and also invariants, so that if a method is ever called with invalid parameters, returns invalid values, or the object gets into an invalid state, an error of some kind will be thrown.

Seiber answered 30/7, 2010 at 12:35 Comment(0)
K
0

No is not possible. Testing for side effects is difficult in any testing stage. It is differrent in different projects (product development, tool development, business application development, game development etc.).

Without a complete regression test side effects cannot be found.

In a typical project (I experienced) the question for "does this change have any side effects" is often asked towards the end of the project (close to go live) or when someone wants to add a hot fix in an already productive system. I found out that without a regression test the only (still risky) quality control measure is code review.

Hope it helps.

Kesselring answered 30/7, 2010 at 12:35 Comment(0)
E
0

One technique is to randomize the order of your unit tests. If your test suite is reasonably comprehensive, then randomizing the tests' orders can reveal unexpected side effects, tests which incorrectly depend on previous tests' state, and so on.

Google Test (for C++) can do this; I don't know of other frameworks that have this feature.

Echoism answered 30/7, 2010 at 12:43 Comment(0)

© 2022 - 2024 — McMap. All rights reserved.