“Conventional xUnit wisdom” holds that one should organize tests into groups and implement these tests in one class, factoring common set up and tear down operations into the class’s setUp() and tearDown() methods. I think this wisdom is perpetuated mostly by the apparently intentional design of xUnit testing frameworks for this kind of usage. (Note that Boost.Test is fundamentally different, and people absolutely hate it when they first encounter it just because of this, even though it seems to work better in the long run for reasons we shall shortly see.)
This pattern quickly becomes problematic. For the first few quick hack tests, it works wonderfully. As the code under test evolves, you’ll usually add small batches of one, two or three tests each.
As you add the sixth test to a class, you notice that you did a pretty similar kind of set up for two tests. You can’t move this code into setUp(), since the other tests in the suite have the opposite precondition. So you factor this code out into a separate private method and call it in both places.
As you write your tenth test, you notice that checking the result from a particular method under test requires three repeated lines of code for each test. Being a diligent coder, you factor this out into an assertFoo() method and replace all the existing instances of the repetition.
As the design evolves, your class under test becomes one of two, then three, different strategies for dealing with a problem. As you start your third test suite, and start retyping the suite of methods you’ve written for testing this sort of class, you think, “This is getting crazy – I need to reuse those methods.” At this point, the first thing you think is, “Let’s create a superclass for this set of test cases, and pull up all these methods we’ve written so they can be reused.
Of course, this is usually a bad design practice, even outside of our xUnit test suites. But it is still a common one, because aggregation is just so darned painful in most common, modern, production-ready languages. In terms of design and reuse, it’s all downhill from here.
Why do we keep getting ourselves into this trap?
xUnit frameworks enforce the use of classes for structural grouping
I’m sure we probably all remember that “neato!” feeling we had when we first saw an xUnit suite and realized that the framework was utilizing reflection or some other language trick to run all the tests. “Hey, that’s pretty cool, and it saves a lot of work!” Well, sure it does.
It may not dawn on us right away that this is not what classes are for!
Normal object oriented design principles suggest that classes are for encapsulation and code reuse
…not for structural grouping of similar objects (in this case, tests), or to provide mechanisms to run lists of objects! We could say that our xUnit suite runners are a historic record of some shortcomings of Smalltalk and Java.
These two uses of classes are at cross-purposes
Showing that these two design patterns are not at cross-purposes would require showing that any logical structural grouping of tests would coincide with (or could be made to coincide with) useful patterns of encapsulation. This seems like a proof too difficult to attempt; however, I can say that my own experience suggests that logical structural grouping of tests does not coincide with useful patterns of encapsulation.
The xUnit Design Heuristic
Stated formally:
Since xUnit’s use of classes to group tests is in conflict with the use of classes for encapsulation and reuse in object oriented design, regard the use of test suites for reusable code or encapsulation as a code smell and prefer creating a driver class.
The driver class encapsulates all or part of a test run and serves as a target for Move Method for those assertions and helper methods. These classes should follow a number of design principles:
Make the driver class own all references to the class under test, fake and mock objects, and other state needed to run the test.
This is simply restating that our driver class’s purpose is encapsulation.
Defer construction of the test’s state to the last possible moment
For our tests to be thorough, they need to be able to test our class under test under many different preconditions. Setting up a lot of state in the class’s constructor may make it difficult or time-consuming to override later. Consider making the various methods set flags or other fields indicating how the test should be run, and having a runner method (or even method object) use these flags for running the test.
Use Builder Pattern to construct test specifications
If designed well, this can greatly improve the readability and the re-usability of the testing code:
void test_Does_not_indicate_balance_receptive_if_balance_return_flag_is_N()
{
BasicRequest()
.With(FID_CARDID).Of("MAS")
.With(FID_BALANCE_RETURN_CAPABLE).Of("N")
.SendsMessageWhere()
.SPIField().Substring(1,1).Equals("N");
}
