The 737 Max disaster

When aircraft crashes aren’t accidents

Over the last few months we’ve witnessed several aviation disasters. From ten thousand feet they look like unfortunate accidents. Everyone follows the rules and disasters happen anyway. Nothing could be further from the truth. If the next few paragraphs make your hair stand on end when you think about the aircraft that nearly took you to the seaside for a summer swim, I’m sorry. An analysis of the possible causes is a chilling reminder that a system that was safe to start with can gradually lead to disaster through a series of isolated decisions that each make sense locally. That’s drift into failure at its best.

While pursuing success in a dynamic, complex environment with limited resources and multiple goal conflicts, a succession of small, everyday decisions eventually produced breakdowns on a massive scale. We have trouble grasping the complexity and normality that gives rise to such large events.

Drift into Failure, Sidney Dekker

How the system began to erode

Image could not be loaded. Norebbo illustration comparing Boeing 737 MAX and 737 aircraft

↗ Medium

Norebbo.com

Did you notice any differences between the 737 Max and the 737 in the picture? Look more closely at the size and position of the jet engines. The 737 Max engines are bigger. A hell of a lot bigger, to be precise. The original diameter was 100 cm. The new one is 176 cm.

They’re so big they couldn’t hang under the wing because there wasn’t enough clearance from the ground. A quick detour. Why does the 737 Max have a bigger engine? Because we all want to fly on holiday and around the world for about the cost of a big supermarket trip. A bigger engine means greater efficiency and less fuel, which brings us back to the price. So we’ve got a big engine that won’t fit under the wing. What do we do? We could move it forward of the wing. Problem solved.

Well, almost. Moving the engine forward of the wing also shifted all the physical forces acting on the aircraft. That makes the 737 Max tend to pitch nose-up when the pilot adds thrust. The more the aircraft rears up, the more it loses the aerodynamic lift beneath the wing that keeps it flying. When lift falls below a critical level, the aircraft starts to fall. At that point the pilot’s only chance is to pitch the nose down, gain speed, and recover the lift they need, assuming they have time to make the maneuver.

Boeing’s aviation engineers thought about the problem and came up with the Maneuvering Characteristics Augmentation System, or MCAS. It monitors the aircraft’s angle, and if it decides an excessively high nose could make the aircraft fall, it does exactly what the pilot would do and pitches the nose down. The aircraft gains speed and everyone is happy again.

A sensor below the cockpit windows monitors the aircraft’s angle. It takes a reading and sends data to a computer. MCAS processes it there and issues commands, which the flight control computer sends digitally over wires to the hydraulics that move the aircraft’s mechanical control surfaces. Originally aircraft had cables and every maneuver was harder because pilots had to fight the aerodynamic forces acting on each control surface. Hydraulics make it easy. A quick detour: the F-105 supersonic light bomber and American workhorse in the Vietnam War also used hydraulics instead of cables, much to the pilots’ displeasure. Unfortunately the main and backup circuits were connected, and a hit that drained the hydraulic fluid meant, at best, losing the aircraft and earning the pilot a visit to the infamous Hanoi Hilton (the North Vietnamese prison). I probably don’t need to describe the worse outcome.

Either way, most aircraft have backups for critical systems. So there are two sensors measuring the aircraft’s nose angle. Unfortunately, and quite illogically, MCAS uses data from only one of them. We can speculate about the reasons, but we probably won’t get to the truth because we can’t see inside the engineers’ heads. If the data is wrong, say because the sensor jams, we have a big problem. MCAS and the flight control computer start to think the aircraft is about to fall because of its angle and push the nose toward the ground. Modern aircraft flight control computers are programmed to keep pilots from making fatal mistakes. If MCAS thinks the aircraft is at a dangerous angle, the pilot can’t do anything about it, even if they pull the yoke clean off. The only way out is to turn MCAS off.

Um, turn it off. Easier said than done. I’m no pilot, but here’s my naive picture of the situation. A quiet flight over the Mediterranean, the aircraft level as a ruler, me flirting with a flight attendant and sipping duty-free Jack Daniel’s. Then someone sends the nose of my aircraft toward the ground. I yank back on the yoke and nothing happens. I look over at the other pilot, who looks back at me just as helplessly. There are now two of us staring helplessly. Then I remember I’m flying a 737 Max and it must be the MCAS they just added. I happen to know where to turn it off. I do that and pull out at the last second, before the nose of the aircraft smashes into the water somewhere near Heraklion.

If you’re wondering how I could be a pilot and know next to nothing about MCAS, here’s a hypothesis. The 737 entered service in 1967. If I got my 737 certification half a century ago, with some training I can fly the Max today even though it’s a completely different aircraft. How can the 737 and 737 Max count as the same aircraft? Well, getting a new aircraft type certified costs a bundle. That’s before training pilots on the new type. Who’d pay for that? The airlines or the companies operating scheduled or charter flights. We aren’t naive, and we’ve come back around to ticket prices. That’s why Boeing is doing everything possible and impossible, and I suspect operating in a gray area, to keep the 737 Max certified as the 737 from 1967. How does it get away with that? Good question.

Here comes one of the biggest acts of chutzpah in our story. For good reasons, aviation separates the role of the people who make aircraft from that of the people who check that every safety procedure is followed. Think of the latter as auditors or inspectors, some sort of bureaucrat. As money poured into aviation, it got harder to keep those bureaucrats in their posts because private aviation companies could pay them a fortune. So it was decided that aviation companies could do the job themselves. A kind of internal audit. Apparently it’s called a Designated Engineering Representative, or DER for short. In Czech I’d rename it KZ, for kozel zahradníkem, a goat put in charge of the garden.

Time for a quick recap of how our system, the 737 Max, drifted into disaster.

• We all want to fly cheaply.
• An aircraft gets bigger engines to make it cheaper to operate.
• The engines are so big they have to move forward of the wing.
• Moving the engines forward changes how the aircraft flies. Adding thrust makes it more prone to pitching up and consequently stalling.
• A new system, MCAS, is created to prevent a stall.
• The aircraft is controlled electronically, unlike in the old days with cables or hydraulics that a person could overpower.
• The aircraft’s control system is software, and MCAS is part of it.
• Software has bugs.
• MCAS uses only one angle sensor even though there are two.
• MCAS takes precedence over the pilot. It can only be deactivated.
• Pay disparities arise between the people making the aircraft and the people meant to inspect them.
• The certification system was outsourced to the company making that same 737.
• We have an aircraft certified as a 737 even though it has nothing in common with the original 1967 version.
• The circle closes

I know few better examples of a system drifting toward catastrophic failure. Notice that each individual step is some form of optimization. Add them up and you get a disaster. It reminds me of the Challenger shuttle disaster and the faulty O-rings in the SRBs, but that’s for another time. This article draws on Gregory Travis’s How the Boeing 737 Max Disaster Looks to a Software Developer, which describes and explains everything in much more detail. I highly recommend it. Agile pioneer Mary Poppendieck looks at the software bug from another angle in What If Your Team Wrote the Code for the 737 MCAS System?. Finally, I’ll mention Drift into Failure: From Hunting Broken Components to Understanding Complex Systems once more, since it addresses how failures happen.

Earlier on Dagblog

• Failure is inevitable, and the flap of butterfly wings that deleted our database