Test Principles - Part I

the fool who invents everything imaginable, motivated by pride and greed only while lacking humility, is a brother in mind with Icarus, doomed to annihilate human kind.

Harmlezz

One goal of writing tests is to detect bugs in production code. To reduce duplication of code and improve readability tests often delegate tasks, which are complex or formulated on a lower abstraction level than the test, to test helper methods. These helper methods have a good chance to be reused by other tests.

Structuring tests in such a way introduces new risks. Different tests may call the same helper method without satisfying the expected pre-condition, which is seldom checked explicitly. Or the helper method is buggy itself, not producing a correct result or state change as expected by the caller.

The question is, how to deal with this problem.

We won't write tests, testing our tests

I would like to explain my implementation approach using Design by Contract for test helper methods. It has the following positive effects:

• early detection of bugs in test code
• and hence less bugs in test code
• improved documentation of what must be provided before using a test helper method
• and what can be expected as a result

One aspect of Design by Contract is to check the pre-conditions of a method before executing its body. In addition a method does check its post-conditions before returning to the caller. Some of the conditions to check can be moved from the method code to unit tests. This shift can be attractive if:

• the readability of the method code would suffer from code, checking conditions
• the complexity of the code, checking conditions, is high
• the code, checking conditions, has a significant impact on the method performance

Post-conditions depending only on input parameters, a returned result or an object state, observable by querying its interface, are good candidates to be moved from the method code to unit tests.

Consider a class Mail capable of being tagged with many labels. The mail class exposes a method tag(String label) which adds a tag with given label to the mail.

One test scenario is to tag a mail with a label, the mail is not already tagged with:

At the first line the test creates a randomly generated mail already tagged with a set of randomly generated labels, returned by the method anyLabels(). The method anyLabels() guarantees that the returned set of labels is never empty. If the scenario for a mail not yet tagged with a label is of importance, a separate test enforcing this condition should be written instead of waiting that the method anyLabels() does return an empty set of labels by chance.

So, the first test helper method that has to guarantee a post-condition, to never return an empty set of labels, has shown up. The code below shows how i do start to implement it:

The aspects of importance to me are the following:

• define the variable, to be returned as result, as soon as possible
• omit the code necessary to provide the promised result
• define the post-conditions and return the yet incomplete result
• omit those post-conditions which would introduce a high complexity bearing the risk of being buggy itself or would obfuscate the code instead of improving its readability.
• use Java assert instead of JUnit Assert to distinguish test code bugs from bugs in the code under test

I always try to run the test using the incomplete helper method before implementing its functionality, just to verify that my post-condition is significant. I skip providing the code for the anyLabels() method required to pass the post-condition, because it does not add anything important to explain my approach.

There is one last thing i like to notice about defining post-conditions that may make a difference in finding test code bugs. Let's consider the test helper method anyLabelNotContainedIn(Set<String> labels) that guarantees to return a label different from any label contained in the set of labels given as input parameter.

The definition of the pre-condition is straight forward while checking the input parameter labels to not contain null is technically optional. It does not prevent the method from generating a proper label but could detect a flaw in the test code.

The important fact is the post-condition is defined in terms of the result variable and the given input parameter only. Whenever possible i suggest to prevent using values held by local variables, because they may have the expected values but their relationship to the returned result is observable only when inspecting the method code which may be buggy. This would compromise the purpose of the post-condition, checking that the result has the promised properties regardless of the way it was calculated.

You should benefit from this approach due to:

• a reduction of undetected bugs in your test code
• a better documentation what you can expect from test helper methods
• an early response when using a test helper method in a way it was not intended for
• improved readability of your tests because drilling down can be reduced to inspecting the first helper method and its pre- and post-conditions
• distinctness of the cause for failing tests due to the usage of Java asserts for test code and JUnit Asserts for code under test

Continue reading Part II of my blog series Test Principles, dealing with how to improve the readability of tests.