Lessons from putting DevOps into practice

Yesterday I had the rare opportunity to speak alongside Ladislav Prskavec, Adam Surak and Lukáš Křečan about our experiences adopting DevOps. Although we prepared our talks independently, we agreed on all the main points, regardless of cloud or bare metal, team size or programming language. Here are my own observations and experiences, with a few contributions from those DevOps veterans.

My first encounter with DevOps

I remember quite clearly when I first encountered DevOps. Over Easter 2011 there was a major outage in the AWS region on the US West Coast, where all of GoodData's production systems ran. Our two operations engineers, Vlasta Holer and Kosťa, spent several days working nonstop to migrate production or bring it back up. Migrating to the US East Coast region seemed like an elegant idea until we realized that most other companies had thought of the same thing. If memory serves, we had to start up the machines and bring them online one by one. Back then we had very little automation and very limited help from developers, because it wasn't their responsibility. I remember that Vlasta kept an enormous white tub on his desk, right next to his taekwondo sword. It was full of caffeine capsules and got noticeably emptier over Easter. After days of nonstop firefighting, the guys were exhausted and may even have handed in their notice. It was a turning point. Our head of engineering, Jaroslav Gergic, banged his fist on the table a few weeks or months later and said we were going to do DevOps.

I can't remember whether he read about it on a blog, attended a conference or got the idea somewhere else, and it probably doesn't matter. He deserves a lot of credit for it. It wasn't fashionable then. It was necessary, because GoodData was gaining customers. The volume and complexity of the data grew, as did expectations about speed, in direct proportion to what customers were willing to pay us each month. That's life in software as a service. Your customers don't care that the cloud provider underneath you has problems. They're paying you, and you have a problem. SLAs give them leverage when it's time to negotiate a contract renewal.

You can get off to a good start with DevOps or a bad one. Ours wasn't a good or particularly fortunate one. The word came down that we'd be doing DevOps from now on. Nobody had experience, and we jumped headfirst into the unknown. My first encounter with a brick wall came when Lukáš Křečan and I wanted to put MongoDB into production. We asked Jaroslav Gergic how we were supposed to do it in this new DevOps setup, and he told us to use Puppet, which we'd just started using for infrastructure automation. I knew a little Java and had grown up on Windows, as a user of course. Suddenly I had to run a database on Linux and do it all through Puppet, about which the most I knew was that it was written in Ruby. Lukáš Křečan wasn't much better off, but at least he'd read some academic paper on what CAP was. That we'd actually be running a distributed system, that we'd have to put the logs somewhere and that we ought to monitor it only dawned on us much later. These are exactly the things you don't usually learn as a developer when someone else runs everything. To this day, I'm immensely grateful to Jaroslav Gergic and give him credit for the experience I gained.

Two worlds working together

Development and operations, or ops as we sometimes call it, are two different worlds. The work and the knowledge it draws on are different.

Ops people usually don't have a detailed picture of the application running on the infrastructure they operate, and developers usually don't have a detailed picture of the infrastructure their application runs on.

Ops engineer

Linux
The kernel and its internals
Networks, firewalls and security
Monitoring, Nagios and Munin
Ansible, Salt, Puppet, Chef or another configuration management tool

Developer

Windows, with honorable exceptions
Java, PHP, JavaScript or another language
Design patterns
Tests
Relational databases

Silos emerge naturally. There's us developers and there's operations, with a deep ditch in between. DevOps fills in that ditch and removes the barriers that keep development and operations from working together. It's about getting the two worlds to work together and joining up the work they do.

DevOps can take different forms, and there are different ways to introduce it. Here are some things that worked for us and some that didn't.

No ops

A team with no dedicated ops engineers was one of the things Lukáš Křečan mentioned as an advantage at Liftago. He even called his talk “Extreme DevOps.” We started much the same way at Zonky a few years ago. SiteOne ran our infrastructure, including the virtual machines, backups and basic monitoring. The five developers we had took care of the rest. As the company grew, that became unsustainable and we brought in our first operations engineer. Looking at the nature of the two roles, development and operations, you can see that No Ops requires extremely experienced developers with a lot of operations experience. Even then it only works up to a certain size, whether measured in the company, data, customers, developers, infrastructure or whatever else generates operations work. Even after moving into the cloud, where we're an AWS shop, we still have a lot of automation, monitoring and infrastructure work that would eat into our development capacity.

I agreed with Adam Surak that it's hard to find developers with operations experience. In interviews Adam asks roughly how an HTTP connection is established and what goes on during that process. I ask a simple question about how to get telemetry out of an apparently stuck JVM. If a developer knows there's such a thing as a thread dump, they want to trigger it using GUI tools that typically aren't available. More than 5% of the developers I've ever interviewed have no idea there's a QUIT signal or where to find its output. When it comes to operating distributed systems, shared state, split-brain scenarios and data consistency versus availability, maybe 1% know their way around it.

Lukáš Křečan and Ladislav Prskavec pointed out, and I basically agree, that No Ops works when you can afford to buy part of your infrastructure as a service. Liftago and Apiary buy MongoDB and RabbitMQ this way. These are distributed systems you really don't want to operate yourself, because that's not how you make your money. We all agreed that it makes no sense to build infrastructure you can buy for very little. Lots of services came up that the companies had in common, including PagerDuty, LaunchDarkly, Splunk or Papertrail, Datadog and others.

Everybody does everything

This is a very bad arrangement. If you dump all the difficult operations work on excellent developers who have little ops experience, you'll just frustrate them. DevOps definitely isn't about everyone doing everything. At GoodData we had product teams or teams responsible for services, each with dedicated ops engineers. Ladislav Prskavec pointed out that operations tasks are different from product development. Product development is time-boxed, and we try to write stories of roughly equal complexity. In operations it doesn't make much sense to follow Scrum strictly, especially time-boxing, and the work is better suited to Kanban. Liftago does manage to fit operations work into sprints by leaving one or two days of spare capacity in each sprint. At Zonky we have dedicated capacity in a special team that deals with complex technical problems, such as rewriting something for Spring Boot, as well as an SRE team that looks after infrastructure.

Developers in production

Unsurprisingly, we agreed that developers need to take an on-call shift at least once in a while. The main benefit is a change in attitude, from “It's not my problem” to “It's my problem too when being on call wakes me at two in the morning.” Developers pay more attention to everything that makes the application easier to operate and understand, including logging, tooling and monitoring. Adam Surak mentioned that someone can be excused from on-call duty if, for instance, they have trouble sleeping. I agree. Developers need to be properly prepared and trained before going on call. At Apiary that takes six to eight weeks, and there should always be someone more experienced to escalate to. You don't have to start in a front-line trench. You can work up to it.

Apiary has runbooks, and at Zonky we have cookbooks. These are sets of commands or instructions for what to do in response to a particular alert or for a particular operations task.

Archive image in Lessons from putting DevOps into practice, figure 1

At Zonky, developers are ready after around six months. They can try some of the operations work in nonproduction environments, which they support while serving on what we call the sweep team. Developers on call that week, which for us runs from 9:00 a.m. Monday to 9:00 a.m. the next Monday, handle regular Thursday releases. They can rehearse them in an environment that resembles production as closely as possible.

Obsessed with data

Ladislav Prskavec mentioned that his SRE team at Apiary measures things like flaky tests. Virtually all of us measure how long the build and test stages take. We routinely use metrics to uncover manual or inefficient work and put a lot of emphasis on automation. If a manual step takes five minutes and you do it five times a day on four days of the week, that's 100 minutes a week.

Here's an example of a Zonky monitoring report described by Tomáš Vaněk in the article Speeding up the build process by 500%.

Archive image in Lessons from putting DevOps into practice, figure 2

I'd say that the kind of laziness that leads to automation is something developers bring to DevOps.

Culture and a few other observations

We all agreed on the need for a postmortem after every production incident, to find the cause and define concrete steps to fix the problem completely or at least address it in part.

Archive image in Lessons from putting DevOps into practice, figure 3

There was a lot of emphasis on making it safe for people to work without being paralyzed by the fear of making a mistake. Adam Surak said that in his first month at Algolia he caused an outage for the company's biggest customer that cost the company much more than his salary. He managed to do it twice that month.

All the companies test on customers. At Zonky we use an anonymized production database that we load automatically into a test environment each week. It saves a lot of work preparing test data for edge cases. We also use the database for regression tests, for example when optimizing the calculation of investor fees. Liftago, Apiary and Zonky use LaunchDarkly for staged rollouts, and Algolia has its own solution. Bugs have always existed and always will, and often you'll only meet them in production. Even the best prevention can't replace the ability to survive them. You can read more in my earlier article, The unknown unknown.

Adam Surak made three particularly apt observations about adopting DevOps.

Google solves Google's problems. When you have Google's problems, solve them the way Google does. Until then, look at what companies like yours do. Ladislav Prskavec mentioned the book Seeking SRE, which describes the experiences of smaller companies. I'd also recommend Release It! Second edition.
Ninety percent automation is better than no automation. People often get stuck on the idea that if they can't automate something completely, they'd rather do nothing. DevOps adoption is itself iterative. We keep learning how to improve and work more efficiently, taking into account what it'll cost us.
Wait until you’re ready for chaos engineering. Until then, try sending a few engineers on vacation and see what it does to your availability. I agree, and here's an excellent article on the subject, The Limitations of Chaos Engineering.

We said much more over the hour and a half, so I'd recommend watching the videos on the CZJUG SlidesLive channel, which will be available at the end of May. If anything about DevOps at Zonky caught your interest, my colleagues and I will be glad to tell you more.