Testing in production

Photograph of a skydive — Testing in production is like skydiving. If you know what you’re doing, it’s reasonably safe.

In the puritanical world of the traditional waterfall development cycle, the very idea of testing in production is rank heresy. Or is it? Testing in production does not mean skipping testing. It’s a sophisticated approach that uses real data and live users. In the world of agile development, microservices and the cloud, it’s a necessity. Testing itself is only one piece of the software delivery puzzle, so I’ll take a wider look at the subject.

Motivation

First, shorter development cycles and the need for agility make it necessary. In cost terms, it’s about what you invest in development versus what you get back once users get their hands on it. Much product development rests on hypotheses, some better supported by user research and testing than others. You only get to validate them when you turn them into a product and put it in users’ hands.

Suppose my hypothesis is that a loyalty program costing X will bring in Y new customers. The only way to test it for real is to build the program. You can build it in a week, a month or six months, depending on how perfect you want it to be. The longer it takes, the more resources you invest and the longer you wait for users to get it and for you to see whether it works as expected.

So in product development you try to take small steps and validate hypotheses quickly until the return justifies the cost. You test the business proposition with users, and your development cycle has to reflect that. If you need a change at least once a week, the cycle can’t take a month or even two weeks. Your technical testing has to keep pace.

What keeps the development cycle from getting shorter?

When I worked on “boxed software,” it took several months to go from the initial analysis through development and testing to burning a CD. Changing a single bit meant going through half that ordeal again and dealing with logistics if you couldn’t just ship patches. It was a nightmare.

At GoodData in software as a service (SaaS), and at Zonky in fintech, the cycle dropped dramatically to as little as a week. That was possible because of the nature of the product or service and, more importantly, because we controlled the whole chain from design and development through deployment and distribution to the customer. Even then, getting the cycle down to days wasn’t really feasible except when hot-patching production during serious incidents.

Three constraints stood in the way. Each was a problem on its own, and worse, they reinforced each other in a closed loop of causes and effects.

People’s mindset

Most IT managers are former developers accustomed to a particular way of working. It was tailored to boxed software, where changing a single bit cost a lot and each new version brought large changes. The focus was on tests, automated or manual, that would act as a barrier against regressions.

Getting out of that comfort zone was hard. I’d sum it up as “we’ve always done it that way.” Process obstacles could get in the way too. Development was only one piece of a puzzle bounded by regulations or legislation. Change was possible, but beyond the development team’s authority. Marketing, sales, legal and compliance and everybody else involved had to line up. If it was difficult for IT management or even developers, it often seemed like a pipe dream for those departments.

Of course there were other ways to do it. You just had to look around. Examples included Facebook, Google, Netflix and Amazon. Pointing to their services often ended, and still ends, with the objection that “if something breaks there, it doesn’t matter.” The world’s biggest online retailer, Amazon, is enough to answer that. It would matter a hell of a lot if orders stopped going through on Black Friday, for example. It’s the sort of objection you reach for when you’re looking for any excuse.

That mindset, a reluctance or failure to see how to leave the comfort zone from top to bottom, is the main obstacle to shortening the development cycle.

Tests, test types and environments

The types of tests also made shortening the cycle difficult. I’m talking about manual tests, but above all about automated ones. When end-to-end integration tests are the backbone of a suite, they’re slow to run, hard to maintain and often brittle. Nothing is worse than flaky tests that undermine confidence in the entire suite, unless you’re Google and have your own way to handle them.

The tests are expensive to maintain, and so are the shared environments they run in. Shared environments matter because what everybody shares rarely gets looked after. So you create a specialist department or pick owners and make it their problem. The more environments (dev, test, integration, UAT, pre-prod), the more hardware, software licenses and maintenance. At Zonky we used all the public cloud advantages AWS offered, including Spot Fleet instances and scaling for development peaks during business hours. Even then, infrastructure outside development was our biggest cost item.

The environments differ from production in size and often in topology. It’s like that recipe from the Czech TV cook Babica: if you don’t have a lime, use a lemon, and if you don’t have a lemon, leave it out. We have a Redis cluster in production, but the other environments use in-memory caching in the JVM because it’s cheaper and easier to manage.

Changing the backbone of the test suite to reduce the number of environments runs up against costs, people’s mindset and the architecture, as the previous and following sections show.

Architecture and size of changes

In a monolithic architecture, every change requires redeploying the whole application. At the end of the cycle all changes get mixed together. The resulting change is enormous, and you have to test practically everything because it’s hard to isolate the impact of the changes and select the matching regression tests. A failure in one part of a monolithic application can bring down the whole application, which is the unit you deploy. That risk calls for more types of tests and in turn more types of environments. We’re back in a vicious circle.

Techniques for testing in production

What makes it possible to move testing into production is the application architecture and the techniques we have for deploying changes and rolling them out, meaning making them visible to users.

Deploy versus rollout

Deploy and rollout used to be one step, with a change visible to every user as soon as it was deployed. Today they are two steps. The new package used to replace the old one, so there was only ever one version. Cloud application stacks such as AWS Beanstalk or Kubernetes offer several deployment strategies . For example, you can deploy a new version and gradually redirect all traffic to it using a canary release or a blue-green deployment.

Diagram comparing a blue/green traffic switch with gradual canary routing — The difference between a blue-green deployment and a canary release

I took the image from the article Deploy and Release: Decouple for Speed and Safety.

If something goes wrong, you can roll back quickly to the previous version, even automatically based on application telemetry such as error rate or latency, or basic tests we run after each deployment (signing in, displaying the dashboard, submitting a loan request, making an investment).

To roll changes out to specific users, you can use feature toggles, or feature flags, which act like switches in the code. The new functionality is inside an if block and only runs if the condition is met. You can therefore target particular changes at particular users.

Diagram showing how feature flags switch features on or off — How feature flags work

At Zonky we used feature toggles so a change went first to Zonky employees, then to a limited group of external users and finally to everyone in stages. We could stop the rollout at any point. For sensitive product changes, we even limited the rollout to certain business hours so we could watch it, and switched it off for everyone outside those hours. We managed rollouts and feature toggles with LaunchDarkly, where the previous and following screenshots came from.

Feature flag overview in the LaunchDarkly dashboard — Feature flag dashboard

LaunchDarkly settings for rolling out a feature by percentage — Percentage rollout

Synthetic accounts and identifying production test data

One technique for testing in production that we used at GoodData and later at Zonky was test journeys , which exercise basic use cases after every deployment or continuously. It’s important to design the system with test data in mind, or at least have it cope with that data through automatic or semi-automatic compensating actions that clean it up.

I remember two stories from Zonky. The tests used real data, and at Zonky that meant real money, so the system had to be ready. When we tested loan requests, the user interface marked the loan as a test so the back office team wouldn’t handle it.

The test loan request had a flag in the database to keep it out of reporting. As bad luck would have it, the reporting team forgot to add that condition to its reports. There was a small panic because it skewed the funnel that tracked the ratio of loan requests coming in to actual loans going out. Management grilled people over that funnel every week. Test requests increased the input with no corresponding increase in output. An automatic process deleted them periodically, but we certainly didn’t contain the side effects on the first try.

Testing investment in loans was a puzzle. There was no easy workaround, except perhaps selling to an institutional investor, which we hadn’t implemented then. Standa Bejkovec used my production account to pick loans at random, try investing in them and then increase those investments. Money disappeared from my account faster than I could top it up, and my portfolio ended up with a few specimens I certainly wouldn’t frame. It bears repeating that when you test services or parts of a system with side effects in production, you need to account for those effects in the design.

Shadowing

Regression testing is simpler for stateless services. You can use what’s called tap compare testing or shadowing, which GitHub used quite successfully when refactoring its merge strategies. In a shadowing test, the old version keeps running while some traffic is mirrored to the new version, and you compare the results. For simplicity, imagine an HTTP service with an HTTP proxy in front of it. The proxy can mirror every GET request to the new version and compare and evaluate both versions’ responses, as Diffy does. In practice it’s much more complicated.

Diagram of mirroring production requests to compare service versions — Traffic shadowing

Are microservices and the cloud game changers? Not quite

You might think microservices and the cloud solve the problems of traditional testing across multiple environments. That’s only partly true. We’ve already discussed the cost of cloud environments. What microservices do bring is a change in testing strategies.

Microservices, if they really are microservices, are much better isolated than monolithic applications. They should have limited responsibilities and be smaller as a result. They have a limited blast radius and can be cycled and deployed quickly. Setting aside their communication networks (the service mesh), their complexity shifts into deployment. That means we need to be able to test them in isolation.

Diagram of the layers of microservice testing — Testing microservices in isolation

Paradoxically, deployment also gets more complex when you use cloud services beyond virtual machine infrastructure. If you use an API gateway or managed databases (AWS RDS, Azure SQL), for example, you need to create, configure and initialize a separate instance for each environment. If microservices use separate databases, that can mean N instances or one larger instance split by schemas or logical databases. One approach or the other will cost more in money or in provisioning complexity.

Most microservice testing happens in isolation, and only a small part happens in the integration environment, if there is one at all. Testing strategy, deployment and operations all need to allow for unexpected and therefore untested situations. A phased rollout, monitoring and fast rollback are requirements. Testing in production is a reality you have to plan for.

One size doesn’t fit all

I wouldn’t recommend testing in production if you don’t control the entire distribution chain. You can introduce it gradually and there aren’t many cases where it can’t be used. In fields where it might be hardest to imagine, like financial services and online betting, it often happens without you realizing it.

Ideally, start with the infrastructure to separate deployment from rollout and experiment with small changes even if you have no microservices or have only partly adopted them. If you’re interested in testing in production, I recommend the exhaustive article Testing in Production, the safe way or the podcast on testing in production we recorded with Zdeněk Merta, Martin Damovský and Jiri Fabian ☢.

Sources and acknowledgments

I spent many hours talking with Zdeněk Merta, who helped me understand the subject in a much broader context. I’d like to thank him and share the books he recommends for further reading. I’d also like to thank Martin Damovský. Besides helping me put production testing into practice at Zonky, he provided valuable criticism and pointed out that the subject is much broader and is starting to appear under the name QA in production, but that’s for another time. Which brings us back to the beginning, because testing in production is just one piece of the software delivery puzzle…