The Phoenix Project: Fiction About How IT Really Works and the Move to DevOps

I’ve just finished an excellent book, Phoenix Project, which tells a fictional story about the move to DevOps and Kanban, one that rings true in real life. The traps and pitfalls Bill Palmer, the main character, has to navigate lay bare the world of IT and how it fails to work. Here are the observations I found most important, with some thoughts of my own.

Intro

The book opens with the rather unexpected appointment of its main character as VP of IT operations at an auto-parts manufacturer. You couldn’t imagine a worse entrance. The company has been missing its targets for some time, losing money, and its stock is falling faster than Jan Fischer’s popularity. Its competitors are kicking its ass, and analysts think it will probably need to be broken up and sold off. As if that weren’t enough, a pack of auditors is circling with a list of security and legal violations long enough to fill more pages than the abduction of Iveta Bartošová. Everyone, from management to marketing to sales, is pinning their hopes on the Phoenix project, which is supposed to help the company meet its goals. It has already slipped several times and run well over budget. Nobody is sure where it stands, and the deployment date is dangerously close. Skeletons fall out of the closet when our hero least expects it, just when you think things can’t get any worse.

A mentor named Erik gradually comes into the picture. He lets Bill see how the factory makes things and work out that Kanban is the way to manage work, and that developers and operations need to adopt DevOps. Another prominent character is Brent, the deus ex machina who intervenes whenever there’s a production incident or some particularly complicated work. Sarah, the head of retail, throws obstacles in IT’s way, as does John, who is responsible for security, for a good while.

Where IT work comes from and what it does

One of Bill Palmer’s first problems is a payroll outage. After a long search, the team works out that someone deployed encryption for Social Security numbers, the counterpart to our Czech personal identification numbers and the laws keeping them confidential, and left invalid data in the database. It gradually becomes clear that the change was never tested because there was no test environment, and that someone slipped it in without going through any change management.

The only thing more dangerous than a developer is a developer conspiring with security

Information security is always flashing their badges at people and making urgend demands, regardless of the consequences to the rest of the organization

The sheer volume of work falling on IT makes Bill’s head spin. To its customers in business, marketing and elsewhere, it looks as though all their projects are late and even the simplest requests take far too long. It soon turns out that those requests account for only a quarter of what IT handles or is asked to do. You need to allow for all of it when estimating a team’s capacity and throughput.

• Business projects. Work arising from the needs of the core business, such as building feature XYZ.
• Internal IT projects. Work IT sets itself so it can function efficiently, such as monitoring infrastructure, developer tools, and collecting and analyzing logs.
• Changes. Applying patches, changing configuration, provisioning new machines and other work arising from operational requirements for the applications and services IT delivers.
• Unplanned work. Urgent requests outside the plan, such as incidents and hotfixes.

These four sources of IT work compete for shared resources. The first thing Bill does, and the first thing you should do too, is plan resources separately for the four areas and identify the constraints that cause them to block one another. In the story, the constraint is Brent, the developer and deus ex machina on whom every important request depends.

Build a wall

Processes are supposed to protect people. We need figure out how to protect Brent.

They put a wall in front of Brent, quite literally in the shape of a manager who handles all incoming requests. People were used to calling or emailing Brent directly, saying that something was urgent and had to be done right now. As a result, Brent did nothing but unplanned work even though the Phoenix project needed him. Three developers work with him to pass on what he knows. They gradually categorize the requests and document how to handle them, until other developers take them over altogether. Brent is then free to devote himself entirely to Phoenix.

Unlike other categories of work, unplanned work is recovery work, which almost always takes you away from your goals. That’s why it’s so important to know where your unplanned work is coming from.

The lesson is that people should receive assignments in an orderly way, from a single source. In the book they use Kanban. If it isn’t on the board, nobody works on it. It’s valuable for people to talk about what needs doing. But nothing in IT or anywhere else ever happened just because you talked about it or sent a group email. Unless, of course, you employ the mythical Japanese engineers Ono-se-to and Samo-se-to, otherwise known as “it’ll happen by itself.” Without one authoritative source of work, you risk doing nothing but unplanned work. You won’t even be able to tell where it comes from or how to eliminate it.

Unplanned work kills your ability to do planned work, so you must always do whatever it takes to eradicate it.

Work in Process

With Kanban, development works much like a factory production line. A task or request goes in and moves through each stage of the line. For new features, the stages might be design, coding, testing and deployment. For provisioning new machines, they might be hardware allocation, configuration, software installation and handover. While a task is on the line, it’s called Work In Process (WIP for short). A task is only done when it leaves the line. The more tasks you have on the line in parallel, the more WIP you have. That’s work that still isn’t finished.

Remember, outcomes what matter — not to the process, not controls, or, for that matter, what work you complete

In our story, the amount of WIP turns out to be why IT can’t deliver anything on time. Analysis reveals that the source of the WIP is Brent. Everything comes back to him, and tasks have to wait until he can handle them one by one. The company’s technical debt also contributes to WIP. In the end, Bill takes the rather drastic step of temporarily refusing new work until the team finishes its current WIP. Work can reopen only for tasks or production lines that do not depend on their main constraint, Brent.

Why it’s important to leave some things undone

Bill quickly realizes that, at the rate new work is arriving, IT will eventually be overwhelmed again and WIP will rise. IT will fall short of its commitments and everything will slip behind schedule. Erik steps in to explain that IT is being asked to do things that don’t need doing. The point is to understand which requests contribute to the company’s objectives.

Alongside Bill, you visit marketing, sales and the other departments to see how the company’s objectives shape their work. That helps him identify what they really need from IT and understand how IT might put them at risk. One major source of tasks is the audit. It turns out that the way financial transactions flow makes it possible to take much of the IT system out of the audit’s scope and reduce the number of tasks.

Remember, it goes beyond reducing WIP. Being able to take nedless work out of the system is more important than being able to put more work into the system. To do that, you need to know waht matters to the achievement of the business objectives, whether it’s projects, operations, strategy, compliance with laws and regulations, security, or whatever

A footnote: the lesson here is that audit findings can be interpreted as flexibly as our constitution.

Big projects don’t work, and the big DevOps finale

Bill eventually concludes that the Phoenix project, which they finally manage to deploy through sheer hard work, isn’t all that necessary for the company to meet its objectives. In other words, all that work and the millions of dollars invested will have gone to waste, along with the company’s hopes for survival. One last glimmer of hope is to put together a team that can quickly build a project to target a sales campaign more effectively. The team includes people from development, operations and the business. What can I say? It’s a happy ending. They deliver it right on time, before Black Friday (the busiest shopping day in the US), and the company achieves its best results in years. It is saved from being sold off in pieces and having IT outsourced to Bangalore.

Big projects usually fail because they cost too much and never arrive on time. What matters is time to market, how fast you can get a product in front of customers. It has to be there. It doesn’t have to be perfect. You try things that might catch on, because until you do, you can’t know how customers will take to them or how competitors will react. All of that counts against big projects.

The book aims to take readers through a change in how IT is organized and how it operates, called the Three Ways.

The First way is all about controlling the flow of work from Development to IT operations.

The Second way is about creating constant feedback loops from IT operations back into Development, designing quality into the product at the earliest stages.

The Third way is about creating a culture that reinforces the value of taking risks and learning from failure and the need for repetition and practice to create mastery

Conclusion

An article about a book can never be as thorough as the book itself. I had to leave out plenty of interesting details and their implications. I didn’t have room to explain how they cut the delivery cycle from nine months to days, an echo of continuous delivery, or why it’s important to leave people with some spare time at work. Perhaps I’ll get to that in another article. I’m tempted to say this book is required reading if you’re serious about DevOps. But that wouldn’t be the whole story. In my experience it also gives a remarkably realistic account of how IT works in most companies.

When IT fails, the business fails. It stands to reason that if IT is organized so that it can win, the business wins, too.