JUnit 4.4, a hammer for tests

How far should we extend what testing frameworks can do? I asked myself that while exploring the recent 4.4 release of the popular JUnit testing framework. It brings two major additions. The first is assertThat. The second, and in my opinion the more interesting, is assumptions and theories.

Neither came from JUnit itself. They were taken, or “absorbed,” as people say now ;-), from the testing tools jMock and Popper respectively.

assertThat

The new assertThat method takes the actual value as its first argument and a matcher (an evaluator?) as its second. The matcher determines whether the assertion holds. If that sounds confusing, the code below should clear it up.

assertThat(something, eq("Hello"));
assertThat(something, eq(true));
assertThat(something, isA(Color.class));
assertThat(something, contains("World"));
assertThat(myList, hasItem("3"));

You can define your own matchers and chain them together.

assertThat(something, not(contains("Cheese")));    
assertThat(responseString, either(containsString("color")).or(containsString("colour")));

The main advantage of assertThat over the original assertion methods is readability, at least in some cases. Another plus is that failure messages are provided for you.

assertThat(responseString, anyOf(containsString("color"), containsString("colour")));
// ==> failure message:
// java.lang.AssertionError: 
// Expected: (a string containing "color" or a string containing "colour")
//      got: "Please choose a font"

Assumptions and theories

Assumptions explicitly state the conditions under which a test has to pass. That might be the range of its input data, whether a database connection is available or whether a system property is set. In other words, they describe dependencies outside the test itself. A theory consists of test inputs, an assumption about when the test can use those inputs, and the test itself.

With a theory a developer can say how the code must behave under specified input conditions. Those conditions let automated tools generate inputs beyond what the developer would have expected, or what would be cumbersome to spell out.

Here’s a short excerpt from David Saff and Marat Boshernitsan’s The Practice of Theories: Adding “For-all” Statements to “There-Exists” Tests.

Traditional unit tests in test-driven development compare a few concrete example executions against
the developer’s definition of correct behavior. However, a developer knows more about how a program
should behave than can be expressed through concrete examples. These general insights can be captured
as theories, which precisely express software properties over potentially infinite sets of values.
Combining tests with theories allows developers to say what they mean, and guarantee that their code is
intuitively correct, with less effort. The consistent format of theories enables automatic tools to generate
or discover values that violate these properties, discovering bugs that developers didn’t think to test for.

The following code from the JUnit 4.4 documentation shows what a theory looks like.

@RunWith(Theories.class)
public class UserTest {
  @DataPoint public static String GOOD_USERNAME = "optimus";
  @DataPoint public static String USERNAME_WITH_SLASH = "optimus/prime";

@Theory public void filenameIncludesUsername(String username) {
    assumeThat(username, not(containsString("/")));
    assertThat(new User(username).configFileName(), containsString(username));
  }
}

The DataPoint annotation defines a theory’s inputs. The test runner feeds the filenameIncludesUsername test method all compatible public variables marked with DataPoint. Compatibility means the variable’s type matches the test method’s parameter type. The assumption is expressed using assumeThat.

Here’s the theory in plain language. Assuming username doesn’t contain a slash, the name of the user’s configuration file must contain that username.

When the test runs, inputs that violate the assumption are silently ignored. That’s what the documentation says, anyway. In my own tests it always caused a failure. The DataPoint annotation may also be a poor fit for generating test data. In that case you can implement a subclass of ParameterSupplier.

The classes for assumptions and theories are in a package with “experimental” in its name, but I find the idea useful. It won’t automatically fit every test. It should be a big help for tests that need a wider range and variety of inputs.