TDD - beginner problems and stumbling blocks

Question

While I've written unit tests for most of the code I've done, I only recently got my hands on a copy of TDD by example by Kent Beck. I have always regretted certain design decisions I made since they prevented the application from being 'testable'. I read through the book and while some of it looks alien, I felt that I could manage it and decided to try it out on my current project which is basically a client/server system where the two pieces communicate via. USB. One on the gadget and the other on the host. The application is in Python.

I started off and very soon got entangled in a mess of rewrites and tiny tests which I later figured didn't really test anything. I threw away most of them and and now have a working application for which the tests have all coagulated into just 2.

Based on my experiences, I have a few questions which I'd like to ask. I gained some information from New to TDD: Are there sample applications with tests to show how to do TDD? but have some specific questions which I'd like answers to/discussion on.

1. Kent Beck uses a list which he adds to and strikes out from to guide the development process. How do you make such a list? I initially had a few items like "server should start up", "server should abort if channel is not available" etc. but they got mixed and finally now, it's just something like "client should be able to connect to server" (which subsumed server startup etc.).
2. How do you handle rewrites? I initially selected a half duplex system based on named pipes so that I could develop the application logic on my own machine and then later add the USB communication part. It them moved to become a socket based thing and then moved from using raw sockets to using the Python SocketServer module. Each time things changed, I found that I had to rewrite considerable parts of the tests which was annoying. I'd figured that the tests would be a somewhat invariable guide during my development. They just felt like more code to handle.
3. I needed a client and a server to communicate through the channel to test either side. I could mock one of the sides to test the other but then the whole channel wouldn't be tested and I worry that I'd miss that. This detracted from the whole red/green/refactor rhythm. Is this just lack of experience or am I doing something wrong?
4. The "Fake it till you make it" left me with a lot of messy code that I later spent a lot of time to refactor and clean up. Is this the way things work?
5. At the end of the session, I now have my client and server running with around 3 or 4 unit tests. It took me around a week to do it. I think I could have done it in a day if I were using the unit tests after code way. I fail to see the gain.

I'm looking for comments and advice from people who have implemented large non trivial projects completely (or almost completely) using this methodology. It makes sense to me to follow the way after I have something already running and want to add a new feature but doing it from scratch seems to tiresome and not worth the effort.

P.S. : Please let me know if this should be community wiki and I'll mark it like that.

Update 0 : All the answers were equally helpful. I picked the one I did because it resonated with my experiences the most.

Update 1: Practice Practice Practice!

Zeist

I don't get the "limitation" part of the question title. It seems to be just "advice". – Kalamazoo

What I meant was whether the problems I'm hitting are fundamental to the methodology or whether they're just problems due to my lack of experience. – Zeist

Then ask that in the title. "limitations" implies that you now know so much about TDD that you've discovered the limits. When the questions are simply "how to do TDD?" – Kalamazoo

Answer 1

As a preliminary comment, TDD takes practice. When I look back at the tests I wrote when I began TDD, I see lots of issues, just like when I look at code I wrote a few year ago. Keep doing it, and just like you begin to recognize good code from bad, the same things will happen with your tests - with patience.

How do you make such a list? I initially had a few items like "server should start up", "server should abort if channel is not available" etc. but they got mixed and finally now, it's just something like "client should be able to connect to server"

"The list" can be rather informal (that's the case in Beck's book) but when you move into making the items into tests, try to write the statements in a "[When something happens to this] then [this condition should be true on that]" format. This will force you to think more about what it is you are verifying, how you would verify it and translates directly into tests - or if it doesn't it should give you a clue about which piece of functionality is missing. Think use case / scenario. For instance "server should start up" is unclear, because nobody is initiating an action.

Each time things changed, I found that I had to rewrite considerable parts of the tests which was annoying. I'd figured that the tests would be a somewhat invariable guide during my development. They just felt like more code to handle.

First, yes, tests are more code, and requires maintenance - and writing maintainable tests takes practice. I agree with S. Lott, if you need to change your tests a lot, you are probably testing "too deep". Ideally you want to test at the level of the public interface, which is not likely to change, and not at the level of the implementation detail, which could evolve. But part of the exercise is about coming up with a design, so you should expect to get some of it wrong and have to move/refactor your tests as well.

I could mock one of the sides to test the other but then the whole channel wouldn't be tested and I worry that I'd miss that.

Not totally sure about that one. From the sound of it, using a mock was the right idea: take one side, mock the other one, and check that each side works, assuming the other one is implemented properly. Testing the whole system together is integration testing, which you also want to do, but is typically not part of the TDD process.

The "Fake it till you make it" left me with a lot of messy code that I later spent a lot of time to refactor and clean up. Is this the way things work?

You should spend a lot of time refactoring while doing TDD. On the other hand, when you fake it, it's temporary, and your immediate next step should be to un-fake it. Typically you shouldn't have multiple tests passing because you faked it - you should be focusing on one piece at a time, and work on refactoring it ASAP.

I think I could have done it in a day if I were using the unit tests after code way. I fail to see the gain.

Again, it takes practice, and you should get faster over time. Also, sometimes TDD is more fruitful than others, I find that in some situations, when I know exactly the code I want to write, it's just faster to write a good part of the code, and then write tests.
Besides Beck, one book I enjoyed is The Art of Unit Testing, by Roy Osherove. It's not a TDD book, and it is .Net-oriented, but you might want to give it a look anyways: a good part is about how to write maintainable tests, tests quality and related questions. I found that the book resonated with my experience after having written tests and sometimes struggled to do it right...
So my advice is, don't throw the towel too fast, and give it some time. You might also want to give it a shot on something easier - testing server communication related things doesn't sound like the easiest project to start with!

Parlous

Answer 2

1. Kent Beck uses a list ... finally now, it's just something like "client should be able to connect to server" (which subsumed server startup etc.).

Often a bad practice.

Separate tests for each separate layer of the architecture are good.

Consolidated tests tend to obscure architectural issues.

However, only test the public functions. Not every function.

And don't invest a lot of time optimizing your testing. Redundancy in the tests doesn't hurt as much as it does in the working application. If things change and one test works, but another test breaks, perhaps then you can refactor your tests. Not before.

2. How do you handle rewrites? ... I found that I had to rewrite considerable parts of the tests.

You're testing at too low a level of detail. Test the outermost, public, visible interface. The part that's supposed to be unchanging.

And

Yes, significant architectural change means significant testing change.

And

The test code is how you prove things work. It is almost as important as the application itself. Yes, it's more code. Yes, you must manage it.

3. I needed a client and a server to communicate through the channel to test either side. I could mock one of the sides to test the other but then the whole channel wouldn't be tested ...

There are unit tests. With mocks.

There are integration tests, which test the whole thing.

Don't confuse them.

You can use unit test tools to do integration tests, but they're different things.

And you need to do both.

4. The "Fake it till you make it" left me with a lot of messy code that I later spent a lot of time to refactor and clean up. Is this the way things work?

Yes. That's exactly how it works. In the long run, some people find this more effective than straining their brains trying to do all the design up front. Some people don't like this and want to do all the design up front; you're free to do a lot of design up front if you want to.

I've found that refactoring is a good thing and design up front is too hard. Maybe it's because I've been coding for almost 40 years and my brain is wearing out.

5. I fail to see the gain.

All the true geniuses find that testing slows them down.

The rest of us can't be sure our code works until we have a complete set of tests that prove that it works.

If you don't need proof that your code works, you don't need testing.

Kalamazoo

Your point about not spending time optimising tests is an eye opener. Thanks. – Zeist

Answer 3

Q. Kent Beck uses a list which he adds to and strikes out from to guide the development process. How do you make such a list? I initially had a few items like "server should start up", "server should abort if channel is not available" etc. but they got mixed and finally now, it's just something like "client should be able to connect to server" (which subsumed server startup etc.).

I start by picking anything I might check. In your example, you chose "server starts".

Server starts

Now I look for any simpler test I might want to write. Something with less variation, and fewer moving parts. I might consider "configured server correctly", for example.

Configured server correctly
Server starts

Really, though, "server starts" depends on "configured server correctly", so I make that link clear.

Configured server correctly
Server starts if configured correctly

Now I look for variations. I ask, "What could go wrong?" I could configure the server incorrectly. How many different ways that matter? Each of those makes a test. How might the server not start even though I configured it correctly? Each case of that makes a test.

Q. How do you handle rewrites? I initially selected a half duplex system based on named pipes so that I could develop the application logic on my own machine and then later add the USB communication part. It them moved to become a socket based thing and then moved from using raw sockets to using the Python SocketServer module. Each time things changed, I found that I had to rewrite considerable parts of the tests which was annoying. I'd figured that the tests would be a somewhat invariable guide during my development. They just felt like more code to handle.

When I change behavior, I find it reasonable to change the tests, and even to change them first! If I have to change tests that don't directly check the behavior I'm in the process of changing, though, that's a sign that my tests depend on too many different behaviors. Those are integration tests, which I think are a scam. (Google "Integration tests are a scam")

Q. I needed a client and a server to communicate through the channel to test either side. I could mock one of the sides to test the other but then the whole channel wouldn't be tested and I worry that I'd miss that. This detracted from the whole red/green/refactor rhythm. Is this just lack of experience or am I doing something wrong?

If I build a client, a server, and a channel, then I try to check each in isolation. I start with the client, and when I test-drive it, I decide how the server and channel need to behave. Then I implement the channel and server each to match the behavior I need. When checking the client, I stub the channel; when checking the server, I mock the channel; when checking the channel, I stub and mock both client and server. I hope this makes sense to you, since I have to make some serious assumptions about the nature of this client, server, and channel.

Q. The "Fake it till you make it" left me with a lot of messy code that I later spent a lot of time to refactor and clean up. Is this the way things work?

If you let your "fake it" code get very messy before cleaning it up, then you might have spent too long faking it. That said, I find that even though I end up cleaning up more code with TDD, the overall rhythm feels much better. This comes from practice.

Q. At the end of the session, I now have my client and server running with around 3 or 4 unit tests. It took me around a week to do it. I think I could have done it in a day if I were using the unit tests after code way. I fail to see the gain.

I have to say that unless your client and server are very, very simple, you need more than 3 or 4 tests each to check them thoroughly. I will guess that your tests check (or at least execute) a number of different behaviors at once, and that might account for the effort it took you to write them.

Also, don't measure the learning curve. My first real TDD experience consisted of re-writing 3 months' worth of work in 9, 14-hour days. I had 125 tests that took 12 minutes to run. I had no idea what I was doing, and it felt slow, but it felt steady, and the results were fantastic. I essentially re-wrote in 3 weeks what originally took 3 months to get wrong. If I wrote it now, I could probably do it in 3-5 days. The difference? My test suite would have 500 tests that take 1-2 seconds to run. That came with practice.

Torose

Answer 4

As a novice programmer, the thing I found tricky about test-driven development was the idea that testing should come first.

To the novice, that’s not actually true. Design comes first. (Interfaces, objects and classes, methods, whatever’s appropriate to your language.) Then you write your tests to that. Then you write the code that actually does stuff.

It’s been a while since I looked at the book, but Beck seems to write as if the design of the code just sort of happens unconsciously in your head. For experienced programmers, that may be true, but for noobs like me, nuh-uh.

I found the first few chapters of Code Complete really useful for thinking about design. They emphasise the fact that your design may well change, even once you’re down at the nitty gritty level of implementation. When that happens, you may well have to re-write your tests, because they were based on the same assumptions as your design.

Coding is hard. Let’s go shopping.

Shouldst

Answer 5

For point one, see a question I asked a while back relating to your first point.

Rather than handle the other points in turn, I'll offer some global advice. Practice. It took me a good while and a few 'dodgy' projects (personal though) to actual get TDD. Just Google for much more compelling reasons on why TDD is so good.

Despite the tests driving the design of my code, I still get a whiteboard and scribble out some design. From this, at least you have some idea of what you are meant to be doing. Then I produce the list of tests per fixture that I think I need. Once you start working, more features and tests get added to the list.

One thing that stood out from your question is the act of rewriting your tests again. This sounds like you are carrying out behavioural tests, rather than state. In other words, the tests sound too closely tied to your code. Thus, a simple change that doesn't effect the output will break some tests. Unit testing (at least good unit testing) too, is a skill to master.

I recommend the Google Testing Blog quite heavily because some of the articles on there made my testing for TDD projects much better.

Francie

The feeling I have right now is something similar to what I felt when I first tried OOP and didn't 'get it'. Perhaps it is just lack of practice and experience. Thanks for your answer and the link to the other question. – Zeist

I'd say it took me about 12 months to feel fully happy with everything. Half of that was on industry projects. Now I literally cannot write a single line of code without TDD, its so addictive. Coding is fun again as they say. – Francie

Answer 6

The the named pipes were put behind the right interface, changing how that interface is implemented (from named pipes to sockets to another sockets library) should only impact tests for the component that implements that interface. So cutting things up more/differently would have helped... That interface the sockets are behind will likely evolve to.

I started doing TDD maybe 6 months ago? I am still learning myself. I can say over time my tests and code have gotten much better, so keep it up. I really recommend the book XUnit Design Patterns as well.

Marika

Yes. One of the things I figured out was my interface problem. I used to a pass a 'from' and 'to' to the server but then later it became just a single channel. – Zeist

Answer 7

How do you make such a list to add to and strike out from to guide the development process? I initially had a few items like "server should start up", "server should abort if channel is not available"

Items in TDD TODO lists are finer grained than that, they aim at testing one behavior of one method only, for instance:

• test successful client connection
• test client connection error type 1
• test client connection error type 2
• test successful client communication
• test client communication fails when not connected

You could build a list of tests (positive and negative) for every example you gave. Moreover, when unit testing you do not establish any connection between the server and the client. You just invoke methods in isolation, ... This answers question 3.

How do you handle rewrites?

If the unit test tests behavior and not implementation, then they do not have to be rewritten. If unit test code really creates a named pipe to communicate with production code and, then obviously the tests have to be modified when switching from pipe to socket. Unit tests shall stay away from external resources such as filesystems, networks, databases because they are slow, can be unavailable ... see these Unit Testing rules.

This implies the lowest level function are not unit tested, they will be tested with integration tests, where the whole system is tested end-to-end.

Willin

Do you have suggestions for writing tests for code which is not fully implemented, and that you do not plan on implementing immediately? – Escort

Don't. Why writing the tests now if you're not writing the code ? If the code is to be written soon, I may just add one-liner comments with the test descriptions, or empty skipped test. I won't bother implementing the future tests now. Why not asking this in a Stack overflow question ? – Willin

1) In my case, I want to get tests around the code that is there. 2) It's more difficult to draft a good question than it is a comment, and I noticed you referenced behavior and implementation in unit testing. Given further thought, it feels like more of a blog post question, as it revolves around strategy. – Escort