A few thoughts on tests

I was asked to share my experience with JUnit 4 and with writing tests using it. I was going to brush it off with a sentence about how easy the library is to use, but then it occurred to me that what matters here is the habits I’ve developed while writing tests. As well as JUnit, I’ll cover what to watch for, how to write tests, and the excellent library Mockito, which makes tests easier to write. I hope everyone finds something for them in the article.

JUnit

I should say up front that I’ve never worked with TestNG, so if JUnit is missing something you need, TestNG may well have it.

When Kent Beck wrote JUnit 4, annotations had been part of Java for some time. In JUnit 3 you had to extend TestCase and include the word test in your method names. I no longer remember if it had to come at the start. In version 4, you only needed to mark the methods containing tests with @Test.

JUnit 4 added other annotations and constructs I started using, though they fall short of perfection. @BeforeClass and @AfterClass handle setup and cleanup before and after the tests in a class. That’s welcome, except they can only be used on static methods, which makes them less useful. I haven’t found a good reason they have to be static.

Adding a more readable assertion method, assertThat, was a good move. It’s basically the only kind of assertion I use now.

import static org.hamcrest.CoreMatchers.is; 
Cart cart = new Cart();
cart.put("something"); 
assertThat("The method size returned wrong count of items in the cart.", cart.size(), is(1));

The snag is that it takes an org.hamcrest.Matcher as its second argument, or third if you supply a failure message, and that comes from a different library. That could be a good thing, except Kent Beck dealt with it in the worst possible way. He took part of Hamcrest and baked it into JUnit, original package and all.

If you want a newer Hamcrest, you’re out of luck. You generally want one when you’re writing your own Matcher and want a better failure message.

A later JUnit 4 release added Assume and its assumeTrue(boolean) method. I was excited about it at first, but that wore off. The idea is to check a prerequisite before a test starts and skip the test when the prerequisite isn’t met. The problem is that you can’t tell whether the test actually ran or was ignored, so the results tell you less.

General advice

Write assertions with a message that puts the failure in context. If the test fails, the results alone should tell you what went wrong and where, without making you hunt for the assertion and work out why. This is particularly useful with several assertions in a method. It also makes tests easier to read and documents why an assertion is there.

assertThat("The method size returned wrong count of items in the cart.", cart.size(), is(1));

Name your test methods to say what you’re testing.

@Test 
public void testCartSize()....

Don’t pack every assertion for a feature into one test method. Split the assertions across methods and name each for the case it covers. That makes the tests easier to read and navigate.

@Test 
public void testCartSize() {
  Cart cart = new Cart(); 
  cart.put(new Item());  
  assertThat("The method size returned wrong count of items in the cart.", cart.size(), is(1)); 
  Cart cart = new Cart(); 
  assertThat("The method size returned wrong count of items in the new cart.", cart.size(), is(1));
}

It should look like this.

@Test 
public void testCartSize() {
  Cart cart = new Cart(); 
  cart.put(new Item());  
  assertThat("The method size returned wrong count of items in the cart.", cart.size(), is(1));
}

@Test 
public void testCartSizeOnNewInstance() {
  Cart cart = new Cart(); 
  assertThat("The method size returned wrong count of items for a new car  instance", cart.size(), is(1));
}

Generally the simpler the test code and the shorter the method body, the better.

Avoid tests that depend on timing. I recently had to fix a test that checked that values were removed automatically from a cache after some time.

cache.put("Foo");
 assertTrue(cache.contains("Foo"));
 Thread.sleep(1000);
 assertFalse(cache.contains("Foo"));

It sometimes failed and sometimes passed, even though it waited a relatively long time. The problem was the first assertion. If the cache’s cleanup thread ran between lines 1 and 2, the test just failed. Set aside the fact that if we’re testing a prerequisite here, assumeTrue would make sense.

Above all, the test configuration, which naturally also covered the opposite case where the cache returns values, should have disabled the cleanup thread altogether. Only that specific case should have enabled the cleanup thread, with the first assertion removed.

Here’s the general rule. If a test isn’t reliable, you’re better off without it. Ruthlessly rewrite tests that sometimes pass and sometimes fail for reasons you can’t identify. There’s no such thing as 99.9 percent success in tests. A test passes or it doesn’t. Saying “it fails sometimes and that’s fine,” which I’ve seen happen, is just fooling yourself.

I strongly recommend starting test method names with test, so it’s clear which are tests and which are helpers. It’s also good to name the test for a class by adding the suffix Test to the class name. Nothing creates more confusion than each developer using their own conventions.

It bothers me that IDEs don’t check this rather established convention, in particular for a missing @Test annotation on a test method. I’ve often celebrated a green test run only to curse later when I discovered some tests hadn’t run at all because I forgot the annotation.

People generally dislike writing tests because it feels hard. A good rule of thumb is that if something is hard to test, the code smells. For a complicated setup, I can recommend the excellent Mockito library.

Mockito

Mockito can create mocks and stubs. So something can look like a duck, quack like a duck, and tell you how many times it quacked.

Item item = Mock.mock(Item.class);
Mockito.when(item.getPrice()).thenReturn(10); 
Cart cart = new Cart();
cart.put(item); 
assertThat(cart.getTotalSum(), is(10));
Mockito.verify(item).getPrice();

As you can see, Mockito is easy to use. Set up an Item stub that returns ten when its price method is called. The final verify, to check that something happened on the stub, isn’t needed here. I’ve only included it as an example. Mockito or a similar framework can make writing tests dramatically simpler. I’m excited about Mockito because it removed a ton of boilerplate I used to need for setup.