Manual tests are for dropouts

Every manual test has two parts: a manual and a monkey.

The monkey takes the manual, runs the test, evaluates it and records the result somewhere. Monkeys are lazy by nature, so our monkey doesn’t feel like taking a banana ten times, peeling it, eating it and seeing whether it drops dead because the banana was rotten. It takes the banana, peels it and declares it edible if it looks “normal.”

We don’t have to substitute a human for our monkey. We can ask why the test isn’t automated. What would we gain?

• An automated test has no working hours.
• An automated test can run in parallel.
• An automated test doesn’t distort the results.
• An automated test needs no human interaction.
• An automated test doesn’t get forgotten.

Despite those clear advantages, the prevailing view is still that manual tests can’t be replaced. To a limited extent that’s true. Most, however, are entirely unnecessary and can be automated. One big drawback of manual tests is how little confidence their results give you, because they’re subject to human error.

Costs

The biggest gap between automated and manual tests is in the thing that matters most: the cost of resources. Whether you measure time, price or person-days, manual tests cost more. The way those costs are spread over time differs too.

Automated tests cost most upfront, when you create them. After that, the costs are practically zero. Manual tests, by contrast, consume resources constantly. The high cost of developing an automated test upfront is the biggest barrier to writing one, above all a mental barrier.

Coverage

With manual tests it’s hard to determine how much of the functionality they really cover. They can exercise the most common use cases, but tell you little about the full scope of what they test. With automated unit and integration tests, analysis tools can show code coverage, or how much of the API the tests exercise. That is especially important when responsibilities for different application layers are split. The tests and their level of coverage document what the API is guaranteed to do.

The challenge

The biggest challenge for automated testing is convincing developers and testers that it’s necessary and worthwhile. A developer who doesn’t see that they’re the first to benefit from an automated test will probably never start writing one.

Motivate developers to write automated tests!

• Make it a rule that a test certifies that the code does what it should. Without it, the code is unacceptable. That helps maintain the productive tension between teams that disagree about whether the code does what it should. Testing at the API level lets you specify the minimum required functionality through three things: the public interface, the test and the implementation.
• Guarantee time to write tests. If a developer estimates the cost of a test, accept the estimate and require the test in return.
• Let developers learn to write tests. Test writing doesn’t come automatically. Encourage developers to share what they know about how to do it.
• Support test writing by adopting testing frameworks and regularly reporting pass rates and code coverage.
• Spread the word by showing how automated tests affect product quality.

Replacing manual tests, an outdated approach for the vast majority of test cases, is one of the biggest tasks facing any software company that wants to improve its products. What matters is getting people motivated and convinced. Successful companies have managed that. For the other dropouts, there are manual tests and no certainty about the quality of the result.

Related articles

• Into the fray: integration vs. unit tests
• Database scenarios for integration and unit tests
• How I introduced continuous integration “at our place”
• How I introduced continuous integration “at our place,” part II: Luntbuild
• Eclipse tips: JUnit support

Elsewhere

• Mock testing: Potemkin villages
• Black-box software testing
• Tester: insult or respected profession? Part 1 (Specification tests)