A grid for tests. Pipe dream or reality?

Do you have a pile of integration and unit tests and wonder how to get results from them as fast as possible? Integration tests with time-consuming setup, such as starting a container or initializing a database, are particularly likely to be the bottleneck. We have plenty of those tests, so naturally I’m interested in how to speed them up.

I recently read Fast integration tests in Spring, which offers a solution of sorts for tests that depend on a database. It has several drawbacks, though. It doesn’t work with Hibernate, and test data is harder to express, among other things. Novoj’s solution speeds things up to a point, but it’s still insufficient.

The way to get a real speedup is to run operations in parallel. Simply run the tests at the same time on different machines. Here’s the idea.

• We have a number of computers whose computing power isn’t fully used. We can build a grid from them.
• We can submit jobs to the grid to run tests and report their results.
• A developer starts a test or a suite from their IDE, Maven or the command line. The job goes to the grid, runs asynchronously and reports the results back.

The grid supplies computing power independent of the developer’s hardware. It lets us run tests in parallel and make effective use of the resources available. Solutions for distributed testing do exist.

• Poll — Do You Need Distributed JUnits?
• Tigris — Distributed Testing

Distributed testing has a few gaps, of course. The main one is infrastructure for running tests. Running them against a fixed tree, a specific revision in source control, is possible. Local changes that the developer wants to test are harder. You’d have to compute their transitive closure and propagate the changes along with the test.

Another option is a two-phase commit, though it may go by another name. Locally, run only the tests that directly verify the change. Commit to schedule a run of the whole suite. If the tests pass, the actual commit goes through. Two-phase commit is supported by TeamCity from IntelliJ.

Distributed tests also take some discipline to write. Tests mustn’t interfere with one another, for instance. A test that can run only once at a time on a given configuration, perhaps because of primary-key collisions, would need metadata or annotations to tell the testing framework how to schedule it. There are probably a few more gotchas like these, and they come back to the infrastructure.

I like the idea of using a grid to run tests. Even with the gaps, I can imagine a relatively simple setup that would work as part of continuous integration.