There’s a principle in some circles that suggest that the full suite of regression tests be run after each build, or at the end of each iteration, or before each release. Typically when people talk about stuff like that, they don’t bother to specify what they mean by “full”, or “regression tests”, or even “the” (these tests, but no others?), so it’s hard to tell whether the suggestion is reasonable or not. When the suggestion is reasonable, it’s founded on the idea that there’s a risk that there might be an important problem in the code or the data it deals with—a problem that automated regression tests, typically at the unit level, could catch.
That’s a nifty notion. I’ve worked on several projects where certain unit tests are deemed to be very important, and rightly so. Sometimes these tests are simple assertions that the code deals properly with unexpected values or exceptional conditions. That sounds like a good risk management approach to me. But why stop at the unit tests? If there is a risk of a serious problem, why not move such assertions right into the code? Some might argue that this might cause a performance hit, but on systems that can perform billions of simple comparisons a second, that overhead might be quite tolerable next to the risk of some failure.
Critical thinking is a heuristic approach to solving problems caused by us forgetting something or making an invalid assumption. Testing is strongly informed by critical thinking, and so is good unit testing. The best programmers I’ve ever seen have been great critical thinkers; they programmed like great testers, recognizing that things can be very different from our preconceived ideas. In some cases, they could sometimes afford to put a different emphasis into their unit testing because the product was exceptionally robust, containing important tests within the running code itself. Unit tests are a great way to mitigate certain kinds of risk, but they’re not the only way.
Hi Michael,
That sounds very like “Design by Contract” (DBC), particularly the way it is
baked in to Eiffel. Did you have that in mind?
I prefer “unit tests” to the “DBC” approach because
– automated testing ensures that the code is exercised; some code may not be hit that often – particularly exception handling code.
– similarly edge conditions may not get exercised so often; unit testing tends concentrate on the edges
– it can be difficult to impossible to write “postconditions” that don’t mimic the body of many methods. It’s much easier to be orthogonal with unit tests.
Hi, Paul…
First off, thanks for writing.
I didn’t have DBC in mind when I wrote the post, and I still don’t, specifically. My principal point is to question the universality of any particular approach, even one that I believe to be swell (as I believe unit tests to be). DBC is an example of an approach that can be a worthy alternative to some unit tests.
The regression testing paradigm as it’s often discussed was more important years ago, when unit tests had fallen off people’s radar, and configuration management tools were poorly understood and rarely used. Now many teams have good configuration management and TDD tests in place, so for them, unwelcome changes are much less frequent than they used to be. That indicates an opportunity to change the testing focus. Now that we’ve mitigated certain kinds of risks, let’s use the time and effort we save to address other risks.
If a particular risk is so darned important that we want to run a unit test for it every time, maybe we want to do exactly that. On the other hand, maybe we want to put a check into the code itself. The principal goal here is to ask the question and answer it based on the situation.
Thanks for the comment!
“If there is a risk of a serious problem, why not move such assertions right into the code?”
I’m not sure you want to move the assertion into code because the assertion typically throws an exception during execution (a slow operation). I think the preferred method is to check the parameters and gracefully return from the method if there is a violation of the contract, hence the DBC comment by Paul.
I don’t necessarily agree with Paul’s first point because it seems to imply that code will rot if not run every so often. I cannot imagine that a block of code will spontaneously change. I do, however, expect the intended use of the class or object will change spontaneously and that change should trigger the automated testing.
Hi, Paul…
When you say, “I’m not sure you want to move the assertion into code…”, I want to make it clear that I’m not sure either; I just think it might be a good idea to ask the question. That’s the part of the point of taking a testing perspective when you’re writing code–the daemon in the back of your head that asks “Is there a problem here?” or “Might there be a problem here?” or “If there were a problem here, how might we address it?” A parameter check or a unit test?–the answer is up to you in your context.
Replying to Michael:
I agree with you on the general notion of distrusting “one size fits all” policies, but I’m not so sure of the specifics of converting unit tests to assertions.
Thanks for bringing up the idea, though. It’s got me musing on DBC vs TDD. I may write something contrasting the two approaches.
Replying to Noah:
“I think the preferred method is to check the parameters and gracefully return from the method if there is a violation of the contract, hence the DBC comment by Paul”
We digress a little, but interestingly the DBC ethos is to fail hard with an exception. The idea being to highlight problems early.
“I don’t necessarily agree with Paul’s first point because it seems to imply that code will rot if not run every so often.”
I love that idea! It reminds of my perception of climbing rope: no matter how carefully it is coiled and folded before being put away it seems always to be a tangled mess when it’s next taken out.
Good point, though. What I ought to have said was that putting an assertion in the code is no guarantee that the assertion is exercised during testing.