Is test-driven development a normal approach in game development?

Question

I am just curious since all TDD examples I have seen is web programming related. And if it's not a normal approach, why is it not?

Plasty

Answer 1

TDD has become a favored approach by software developers who are serious about their profession. [IEEE:TDD] The benefits of the approach are significant, and the costs are low by comparison. [The Three Laws of TDD]

There are no software domains for which TDD is inappropriate, or ineffective. However, there are domains in which it is challenging. Gaming happens to be one of these.

Actually, the challenge is not so much gaming as it is UI. The reason UI is a challenge is that you often don't know what you want the UI to look like until you've seen it. UI is one of those things that you have to fiddle with. Getting it right is a deeply iterative process that is full of twists and turns and dead ends and back alleys. Writing tests first for UI is likely to be both difficult and wasteful.

Now before everybody roars off and says: "Uncle Bob says: 'Don't do TDD for UI'" let me say this. The fact that it's hard to do pure TDD for UI does not mean you can't do pure TDD for almost everything else. Much of gaming is about algorithm, and you can use TDD with those algorithms to your heart's delight. It's true, especially in gaming, that some of those algorithms are the kind of code you have to fiddle with, just like UI, and so are probably not amenable to being tested first. But there is a lot of other algorithmic code that can and should be written test first.

The trick here is to follow the single responsibility principle (SRP) and separate those kinds of code that you have to fiddle with, from those kinds that are deterministic. Don't put easy-to-test algorithms in with your UI. Don't mix your speculative code with your non-speculative code. Keep the things that change for reason A separate from the things that change for reason B.

Also, keep this in mind: The fact that some code is hard to test first, does not mean that this code is hard to test second. Once you have fiddled and tweaked and gotten the code to work just the way you like, then you can write the tests demonstrate that the code works the way you think. (You'll be surprised at how many times you find bugs while doing this.)

The problem with writing tests "after the fact" is that often the code is so coupled that it is hard to write the kinds of surgical tests that are most helpful. So if you are writing the kind of code that is hard to test first, you should take care to follow the dependency inversion principle (DIP), and the open/closed principle (OCP) in order to keep the code decoupled enough to test after the fact.

Sebrinasebum

This post clearly and accurately pinpoints the problematics of the TDD approach, and its evangelists. It is seen not as a helpful tool to solve some of the problems faced in software development, but an omnipresent set of cookbook "principles" that must be applied at all cost, everywhere. – Candidate

Agreed Rune, though I think we all have a lot to learn from UB's SOLID principles I am often turned off by the inflexible way that he and his followers deliver their sermons. – Shopworn

I like to call that the cargo cult tax. These are dogmatic approaches to design. Really all TDD is about is tight feedback loops to help design a system. Dogmatically pulling things apart just because it "violates" SRP is short sighted. OCP is yet another security blanket, my test suite allows me to change quickly and know if I broke anything without having to use inheritance/composition. DI is there for us guys that deal with statically typed languages. That is even misused. Why do we go through the ceremony of extracting an interface when we can DI a delegate to make it testable? – Shithead

"TDD has become a favored approach by software developers who are serious about their profession". Fallacy. Five years later, Uncle Bob has changed his mind about TDD and Professionalism. I think TDD has become a favored approach by people who make a living teaching TDD. – Kokura

@donsenior If you think the article you linked shows Uncle Bob changing his mind, I think you need to go and reread it... I don't believe that TDD is a *prerequisite* to professionalism... it currently plays a significant role in professional behavior... it will play a much greater role as we look into the future. He then compares programmers not doing TDD to doctors not washing their hands, and closes with today, at this moment in our history, TDD is not the prerequisite of professionalism that I believe it will become. – Nicknickel

@CamJackson exactly, he has changed his mind, because before he has said (a lot of times, i.e 2007 debate with James Coplien) that TDD WAS a prerequisite to professionalism. But that statement has bring a lot of controversy with a lot of very professional programmers that doesn't do TDD (like the ones he mention on that post). So, yes, he changed his mind. – Kokura

@donsenior Well, whether or not his views have changed slightly over the years, it seems to me that he's still pretty dead set on TDD being an important part of a professional programmer's tookit. And to pick on one of his anti-TDD examples, there are some would describe DHH as less than professional. – Nicknickel

@CamJackson I agree, but the point is, Uncle Bob now admits that you don't necessarily need to do TDD to be serious about your profession. And about the codon video, it's a very good example of an ad hominem fallacy... – Kokura

Answer 2

The simple answer is "no", TDD is not a normal approach in game development. Some people will point at Highmoon and Neil Llopis as counter-examples, but it's a big industry and they are the only people I know of who have fully embraced TDD. I'm sure there are others, but they are the only ones I know of (and I've been in the industry for 5 years).

I think a lot of us have dabbled in unit testing at some point, but for one reason or another it hasn't taken hold. Speaking from personal experience it is hard for a games studio to switch to TDD. Usually a codebase is kept from project to project, and applying TDD to a large existing codebase is both tedious and largely thankless. I'm sure that eventually it would prove fruitful, but getting games coders to buy into it is difficult.

I have had some success in writing unit tests for low-level game engine code, because this code tends to have very few dependencies and is easily encapsulated. This has always been testing after the fact though and not TDD. The higher-level game code is usually harder to write tests for because it has far more dependencies and often is associated with complex data and state. Taking AI as an example, to test AI require some kind of context, meaning a navigation mesh and other objects in the world. Setting up that kind of test in isolation can be non-trivial, especially if the systems involved weren't designed for it.

What is more common in game development, and I've had more personal success with, is smoke testing. You'll often see smoke testing used in conjunction with continuous integration to provide various kinds of feedback on the behaviour of the code. Smoke testing is easier because it can be done by just feeding data into the game and reading back information, without having to compartmentalize your code into tiny testable pieces. Taking AI as the example again, you can tell the game to load up a level and provide a script that loads an AI agent and gives it commands. Then you simply determine if the agent performs those commands. This is a smoke test rather than a unit test because you are running the game as a whole and not testing the AI system in isolation.

In my opinion it is possible to get decent test coverage by unit testing the low-level code while smoke testing the high level behaviours. I think (hope) that other studios are also taking a similar approach.

If my opinion of TDD sounds somewhat ambiguous that's because it is. I'm still somewhat on the fence about it. While I see some benefits (regression testing, emphasis on design before code), applying it and enforcing it while working with a pre-existing codebase seems like a recipe for headaches.

Trampoline

I think what you mean is system testing ("testing conducted on a complete, integrated system") rather than smoke testing ("the first test made after assembly or repairs to a system, to provide some assurance that the system under test will not catastrophically fail"). (Quotes taken from the respective Wikipedia articles.) – Hodgkinson

Answer 3

Games from Within has an article discussing their use of unit testing, the limitations of unit testing with regards to games in particular, and an automated functional testing server that they set up to help with this.

Eteocles

Answer 4

If you are referring to the practice of writing and maintaining unit tests for every bit of code, I'd venture a guess and state that this is not in widespread use in the gaming industry. There are many reasons for this, but I can think of 3 obvious ones:

• Cultural. Programmers are conservative, game programmers doubly so.
• Practical. TDD does not fit very well to the problem domain (too many moving parts).
• Crunchological. There's never enough time.

The TDD paradigm works best in application domains which are not very stateful, or at least where the moving parts are not all moving at the same time, to put it colloquially.

TDD is applicable to parts of the game development process (foundation libraries and such) but "testing" in this line of work usually means running automated fly-through, random key testing, timing io loads, tracking fps spikes, making sure the player can't wriggle his way into causing visual instabilities, stuff like that. The automaton is also very often a humanoid.

TDD can be a useful tool, but its status as a silver bullet that must-be-ubiquitous-when-making-a-system is rather questionable. Development should not be driven by tests, but by reason. RDD is a crappy acronym though - it won't catch on. ;)

Candidate

Answer 5

Probably the main reason is that TDD is preferred by those with languages more conducive to it. But apart from that, games themselves are a poor match for the paradigm anyway.

Generally speaking (and yes, I do mean generally speaking, so please don't swamp me with counterexamples), test-driven design works best for event-driven systems and not so well for simulation-style systems. You can still use tests on your low-level components in games, whether test-driven or just plain unit testing, but for more higher level tasks there is rarely any sort of discrete event that you can simulate with deterministic results.

For example, a web application typically has very distinct inputs (an HTTP request), changes a very small amount of state (for example, records in the database), and generates a largely deterministic output (for example, HTML page). These can be easily checked for validity, and since generating the input is simple it's trivial to create tests.

However with games the input may be hard to simulate (especially if it needs to occur at a certain point... think of getting past loading screens, menu screens, etc.), the amount of state you change may be large (for example, if you have a physics system, or complex reactive AI) and the output is rarely deterministic (random number use is the main culprit here, though things like floating point precision loss is another, as might be hardware specifications, or available CPU time, or the performance of a background thread, etc.).

To do TDD you need to know exactly what you expect to see in a certain event and to have an accurate way of measuring it, and both of these are difficult problems with simulations that avoid discrete events, deliberately include random factors, act differently on different machines, and have analogue outputs such as graphics and audio.

Additionally, there's one massive practical issue which is process startup time. Many of the things you will want to test require the loading of large quantities of data, and if you mock up the data you're not truly testing the algorithm. With this in mind it quickly becomes impractical to have any sort of test scaffolding that just performs individual tasks. You can run tests against a web server without having to take the webserver down each time - that's rarely possible with games unless you do the testing from an embedded scripting language (which is reasonable, and does indeed take place in the industry).

For example, you want to add volumetric shadow rendering to your in-game buildings. So you'd need to write a test that starts up all the necessary subsystems (for example, renderer, game, resource loaders), load in buildings (incl. mesh, materials/textures), load in a camera, position that camera to point at the building, enable the shadows, render a scene, and then somehow decide whether the shadows actually appear in the frame buffer. It's less than practical. In reality you'd have all this scaffolding already there in the form of your game, and you'd just fire it up to conduct a visual test in addition to any assertions within the code itself.

Shipping

Answer 6

Most game developers aren't exactly with it in terms of modern development practices. Thankfully.

But a test-driven development model emphasizes concentrating on how something would be used first, then fleshing out what it does. That in general is good to do since it forces you to concentrate on how a particular feature will actually fit into whatever you're doing (say, a game).

So good game developers do this naturally. Just not explicitly.

Monge

Answer 7

@Rune Once again, please emphasise the 'D' rather than the 'T'. At a unit level, the tests are a thinking tool to help you understand what you want and to drive the design of the code. Certainly at the unit level, I find I end up with cleaner, more robust code. The better the quality of the pieces I put into the system, the better they fit together with fewer (but not no) bugs.

That's not the same thing at all as the sort of serious testing that games need.

Hifi

Can't really see how this contrasts in any way with what I wrote. Thinking about testability when writing code is always good, but it will help you tackle only a subset of the testing needs of a reasonably large-sized game. – Candidate

Answer 8

TDD isn't really a 'normal' approach anywhere yet as it's still relatively new and not universally understood or accepted yet. That isn't to say that some shops don't work that way now but I'm still surprised to hear anyone using it at all at this point.

Shopworn

I'm not going to down vote or anything but TDD is so talked about (regardless if it has true merit or not) that I'm surprised that your "surprised to hear anyone using it at all at this point." – Publish

TDD is pretty much standard in industries like aerospace or medical imaging, and have been for at least a decade. It might not have been called TDD, but automated regression tests, code coverage, etc.. were there long before it became trendy. – Nichellenichol

Not to be nitpicky but automated testing does not necessarily mean that someone is doing TDD. – Jacquenette

m4bwav - People talk about lots of things, that doesn't mean they actually understand or implement them. Kena sorta proves my point. Kena- Regression tests and code coverage != TDD. – Shopworn