Continuous Deployment

I touched on this topic with Filemon and Obrien99 in episode 39 of the podcast, but I'd still like to give it a post of its own. Since I stopped being solely a corporate developer, I've found that there's plenty of inspiration outside the world of enterprise applications. I've already written about NoSQL databases and JavaScript. At GoodData we talk a lot about continuous deployment, which is how to get code we trust into production as quickly as possible. Speed is measured in minutes, rather than months, weeks, days or hours.

We don't make boxed software. You can't buy us like a yogurt, take us home and enjoy us later. We provide a service that users consume simply by clicking in a browser. One advantage is that users don't bear our release costs. Unlike boxed software, we don't have to support deployments across a variety of environments and the problems that come with them, for example.

What continuous deployment is about

Continuous deployment means having the tools and infrastructure to deploy every trusted commit straight to production within minutes. It really isn't as foolhardy as it might look at first. Plenty of internet services do it this way, including Facebook. Unusually, I'll put a few links to excellent discussions of continuous deployment right here in the middle, then add a few observations of my own.

• Continuous Deployment at IMVU: Doing the impossible fifty times a day
• Continuous Deployment at kaChing
• Iterate Like a Whirling Dervish
• Deployment Infrastructure for Continuous Deployment

Tests

Image could not be loaded. Archive image in Continuous Deployment, figure 1

↗ Medium

Do you ever watch rally racing? Those are the maniacs hurtling along a narrow gravel road in purpose-built cars at around 200 km/h, preferably with a thirty-meter drop on either side. Drivers put themselves entirely in the hands of their co-drivers, who read out the pace notes for the stage: the next bend, the profile of the road, danger spots and so on. One mistake and you end up with your feet in the air at best or going in feetfirst at worst.

In continuous deployment, the developer is the driver and the tests are the co-driver. If you can't rely on the tests completely, you end up much as you would in a rally. Reliable tests you trust are the foundation of the whole thing. Of course, they have to be fast too. I remember running integration tests for several hours back at Systinet, where the complexity of deployment made them unreliable. Here's how Timothy Fitz of IMVU describes reliability in continuous deployment.

When I say reliable, I don’t mean “they can fail once in a thousand test runs.” I mean “they must not fail more often than once in a million test runs.” We have around 15k test cases, and they’re run around 70 times a day. That’s a million test cases a day. Even with a literally one in a million chance of an intermittent failure per test case we would still expect to see an intermittent test failure every day. It may be hard to imagine writing rock solid one-in-a-million-or-better tests that drive Internet Explorer to click ajax frontend buttons executing backend apache, php, memcache, mysql, java and solr

Coming from Java, I wondered how anyone could get the tests to run in just a few minutes, given that they run for every commit. There's no simple recipe. You need a bit of everything: running tests in parallel, sharing nothing between tests and mocking. Application design and architecture are closely related. For this kind of testing to work at all, the application can't be written or, heaven forbid, designed like baked Italian pasta.

Infrastructure

Image could not be loaded. Archive image in Continuous Deployment, figure 2

↗ Medium

Original image: Texas Comptroller of Public Accounts

The other key part of continuous deployment is infrastructure that automates the whole process, from a version control hook through monitoring to applying or rolling back the change on the production cluster. I haven't come across anything standardized, if that is even possible. It seems everyone cobbles it together themselves, apart from the continuous integration server.

One good question to end with: Why would I want my changes to reach production as quickly as possible? It comes down to a competitive advantage. The faster we can do it, the faster we can respond to what our business needs. It's about being able to turn things around quickly. Another clear advantage is that you no longer need an expensive quality department, because clicking is automated and QA staff can focus on actual quality.

Culture and people

Culture and the people on a team determine the success of any methodology, and that goes double for continuous deployment. It only works if your people write tests because they genuinely believe in them, rather than because someone told them to. To paraphrase a book title, the motto might be “The test is my co-pilot.”

2 August 2010 — Obrien's take:

• The Case For Continuous Deployment — Goods require shipping. Software should not