Survivorship bias

Do you know the name of the second astronaut on the Moon? Do you know whom Kateřina Neumannová beat in her last race, the one in which she won Olympic gold? Who competed with Roald Amundsen to reach the South Pole? You may know that radio was invented by Guglielmo Marconi, unless the credit eventually goes to Nikola Tesla. I wasn’t trying to test your general knowledge. I wanted to point out that history remembers only the winners. We imitate those winners, often pioneers, in the blind belief that it will help us. That leads to a logical error in judgment called survivorship bias, a type of cognitive bias.

During the Second World War, the Allies tried to reduce aircraft losses. They conducted a study of where planes that returned successfully from combat missions had been hit. They recommended adding armor to the places with the highest concentration of hits. At first glance that makes perfect sense, doesn’t it? Unfortunately, they made a logical error by leaving out the planes that didn’t return from those missions. If they had considered them, it should have occurred to them that the hits on the returning planes couldn’t have been critical, given that those planes made it back. They needed to armor the areas without hits instead. Fortunately, the statistician Abraham Wald spotted the flaw in their reasoning.

I open most of my public talks with survivorship bias. I want my audience to know that whatever I’m describing worked in my context. There may have been thousands of other people or companies doing the same thing and getting exactly the opposite result.

Why am I writing about survivorship bias and this error in judgment? From time to time, we all get carried away by something real or supposed winners are doing and copy it blindly, without considering those who tried the same thing and failed. The point applies whether we’re talking about technology, agile methodologies, processes, programming languages, or company culture. If we want to learn from the winners, we should ask why they do what they do and what conditions they work under.

In my article We’re not Google, and shortcuts like oxygen tents don’t work, I put it this way:

We should pay attention to Google, Netflix, or Spotify, while keeping our feet on the ground and recognizing that their solutions fit the problems they’re solving. We probably all have an intuitive sense that one enormous monolith is wrong. Blindly splitting it into a thousand microservices like Netflix, just because that’s supposedly how things are done, is wrong too.

It was put even better in a panel discussion on DevOps experience by Adam Surak

Google solves Google’s problems. When you have Google’s problems, solve them like Google. Until then, look at how companies like yours solve them.

We should follow the spirit, which can be applied in different contexts, over the letter, which often can’t. That holds whether the context is defined by the type of company (startup or corporation), the technology, the amount of legacy code, the kinds of developers (T-shaped developers, specialists, juniors), or something else. Here are two articles that tied into the subject for me this week.

• Things You Should Never Do by Joel Spolsky, translated by Luboš Račanský
• About When Not to Do Microservices