When intuition leads developers astray: releases and tests

Sometimes in development we let intuition guide our decisions. Unfortunately, it can lead us astray, and sometimes it pays to do the exact opposite. I've picked a few examples of when to resist intuition, mostly around releases and testing.

Release frequency

Problem: Releases are fragile and come with plenty of problems.

Intuition: Release less often. It's safer to expose ourselves to those problems every three months than every month.

It's fairly clear in this case that lengthening the release window won't help much. The problems that come with releasing won't go away. At worst they will pile up. When you release more at once, it's harder to find the cause of a problem. Rollback is another unpleasant issue. The more you release, the more you have to undo and the more complicated it becomes. You also lose every change when you roll back. Intuition is a poor guide here. Do the opposite: if it hurts, do it more often. Most problems with frequent releases come from insufficient automation of the release process or too much reliance on manual testing.

The time manual testing takes should set the release cadence at first. If it takes a week to test the application manually, we should start by aiming for a roughly weekly release cycle. At the same time, we should gradually automate those tests. There is a way to have it both ways and release more often. Some changes don't need manual tests, and we can manually test only the parts of the application that need it. This applies if we are missing only some automated tests or their coverage is insufficient.

The build and release process

Problem: The release or build process must be clear and easy to understand.

Intuition: Let's document the release so anyone can do it.

This is less an argument against intuition than an observation that documentation starts to age as soon as it's written. We should aim to automate the entire process, whether it's a build or a release. The documentation should explain the high-level concept or design, the what, and leave the actual execution, the how, to the automation. As they say in the epic A Song of Ice and Fire, adapted for television as Game of Thrones, words are wind. A piece of code that actually automates the work, however readable it may be, is the one genuine source of truth. Context is another weakness of documentation. Someone who knows a subject inside out will write differently if they remember that readers are missing the context. If you tell someone to restart service XYZ, you assume they know where to find it and how to restart it. They can do that in several ways, from kill -9 to a shell script or the operating system's own tools. Meanwhile, the service, or the person writing the instructions, expects one particular method and never imagines someone would do it differently. That is exactly the kind of context that is hard to capture and depends on the experience of the writer and the reader. In other words, it's shaky ground. That's another reason to rely on automation in place of documentation.

Testing only after implementation

Problem: Testing is expensive.

Intuition: Test at the end so we don't have to repeat an expensive step.

There would be some logic to this if bugs didn't get more expensive to fix the later we find them. Combine that with infrequent releases and it's like sitting on a barrel of gunpowder with the fuse lit. Imagine finding a major bug just before a release scheduled once every six months. Here again intuition reflects a lack of automation in the testing process. The relationship is direct. The more automated tests we have, the more often we get feedback and uncover potential problems. If manual tests are expensive, think about how to reduce the amount of manual testing and balance it with automated tests. Of course, automation comes at a price. Maintaining automated UI tests can be expensive, particularly if the UI changes constantly or there is no adequate abstraction layer, such as a testing DSL, to shield the tests from low-level implementation details.

Testing an incomplete implementation is valuable too. We replace the parts of the system that aren't available with mocks. The implementation is incomplete, but the functionality is there. We can try out certain patterns of user behavior. If we delay testing, we may end up with a fully implemented feature that makes no sense to users or is hard to use. If a test one-third of the way through shows that the feature is useless, we can drop it and invest elsewhere or change it so it makes sense. Tests of any kind are a litmus test for quality and give us feedback that is critical to development.

Flaky automated tests

Problem: Tests often fail.

Intuition: Ignore the results or delete the defective tests.

Flaky tests are a plague. Your intuition may tell you to ignore them, but resist it. When you say a test sometimes fails, the danger is that you won't notice when it starts failing for a real reason. Among other things, that means you need to check somehow that this is the failure you expected. You can also lose track. If five out of a hundred tests keep failing, they may be a different five each time. Ignoring tests makes you less vigilant until you stop paying attention altogether. If some tests are flaky, for example because they depend on third-party services, they need refactoring. Often a poorly designed test is testing the third-party service itself in place of the integration. If the test is well written and still flaky, you can give it a special execution strategy. I think I saw this in a presentation on continuous testing at Google. Flaky tests are run several times. They have a retry strategy that schedules them to run again after an interval.

I've picked a few examples of when it pays to go against your intuition. You'll probably think of others, and I'd be happy to hear about them in the comments.