Canary releases

There are several ways to deliver code to production. Here I'll describe a technique called canary releasing, which lets you deploy changes very safely. We'll look at how it works and what advantages it offers.

The word canary isn't there to amuse the parrots. It refers to a method used in mining. Before gas detectors were developed, canaries served as indicators of harmful gases in newly dug tunnels. The miners let a canary into the tunnel first, and if it survived, the miners went in too. In a canary release, the users, or more precisely a sample of them, take the place of the canary. Let's see how it works. I've chosen an HTTP server as an example, but the basic idea works for almost any stateless service.

Imagine you have a production server and want to deploy a new feature or a patch. You want to do it without the server's users feeling a thing. In other words, you don't want any downtime during deployment or if something goes wrong. Let's assume it's a standard HTTP server. Put a smarter HTTP proxy in front of it. Deploy the new feature or patch to a brand-new server. The proxy automatically forwards requests to the new server when they meet certain criteria, such as the user or type of client. The rest still go to the original server. Once you've verified that the newly deployed code hasn't broken anything, reconfigure the proxy to send traffic to the new server and shut down the original one after a while.

Canary releasing has several advantages.

• A bug can always happen, but it only affects a very small number of users.
• Before switching traffic over, you can run smoke tests to verify that everything is configured correctly.
• You can test things that are hard to simulate in a test environment because of differences in topology or the nature of the load, for example.
• It's easy to roll back. With a conventional release, any problem extends downtime, and rolling back a system where, say, configuration changes have been made is practically impossible.
• You can run A/B tests to see how users respond to a new feature.

One drawback is that you need enough hardware capacity if you release a larger cluster this way. A side benefit is that canary releases are easier to automate. Instead of updating an environment in place and risking interference, you deploy to a clean environment from scratch. At the beginning I mentioned that this works for any stateless service. With services that share state through a database or cache, backward-incompatible changes are a problem. In that case a canary release isn't a good fit.