Failure is inevitable: The butterfly effect that erased our database

Failure is inevitable, and a butterfly’s wings erased our database

I’m listening to Cosmonauts on the Brink of Death by Karel Pacner. Did you know that there was only about a 46% chance Yuri Gagarin’s flight would avoid disaster? This week I saw for myself that failure in complex systems is inevitable, whether they are distributed or otherwise. All you can do is prepare to survive it.

If you look at systems failure theory, you’ll come across the butterfly effect : a small change in input conditions can produce a large change in the outcome. As systems grow more complex, optimizing one part toward a local maximum can cause the entire system to fail. That’s the central thesis of Sidney Dekker’s Drift into Failure: From Hunting Broken Parts to Understanding Complex Systems. I’ll spare you more theory and describe a textbook failure that hit Microsoft and its Azure public cloud in Europe this week.

We use Azure SQL, a database run as a cloud service. The data on physical storage is encrypted with our key, which lives in KeyVault, another cloud service. This is excellent protection if someone tries to make a physical copy of your data or steals the storage medium. It’s called encryption at rest. It has one fairly fundamental property. Lose the key that encrypts the data and you lose all the data, because that same key is used to decrypt it. Microsoft’s engineers based their cleanup procedure on that fact. If Azure SQL can’t connect to the key vault and get the encryption key, an automated process deletes the database within 24 hours. It is deemed worthless because nobody will ever be able to access the data again.

On Wednesday, the Azure region suffered a DNS outage, and our database was in that region. DNS translates symbolic names into addresses for computers on the network. The outage kept Azure SQL from connecting to KeyVault. The automated database deletion process worked perfectly. It didn’t even need 24 hours, just a few minutes. On Wednesday morning, with a flap of a butterfly’s wings, we lost two of our three databases.

I nearly keeled over imagining this happening in production, with the bank losing two-thirds of its data and all the consequences that would follow: downtime, unavailability, the disaster recovery plan, reputational risk, and the media uproar. The database could be replicated to another geographic region, but we still don’t know whether the deletion would have replicated too. We’re still waiting for the investigation results, the postmortem and Microsoft’s response. Once my pulse returned to normal, I realized this was a great opportunity to prepare the whole system for an outage like this and learn to survive it, because failure is inevitable.

The DNS fault was the flap of a butterfly’s wings somewhere in Brazil, and deleting the database was the hurricane it set off somewhere in Uganda. When you look at cloud outages, a single fatal failure is rare. Usually, a series of small failures joins up in a chain that takes down an entire region. In the infamous AWS outage over Easter 2011 , a brief network outage triggered replication of block device backups and took down the whole region.

Key lessons

• The question is when and how the system will fail.
• The system must be robust enough to survive, even if that means degrading some functionality.
• The system must avoid the kind of robustness where optimizing for a local maximum causes a critical failure.
• Prevent failures from cascading through replication
• A failure remains a failure only if we learn nothing from it. Otherwise, it becomes experience.

You might also be interested in

• A public collection of postmortems
• A series by Pavel Toufar on space exploration, He had to lie about the landing. Gagarin couldn’t land in the capsule, and the ejection was kept secret
• Karel Pacner writes books on two of my favorite subjects, espionage and space.

Enjoy the rest of your weekend. If you’d like to get my newsletters regularly, just click Follow on dagblog.cz. You can find the archive here.