Gradual rollout

I've been in this situation more times than I can count. I have tests I trust and I'm absolutely convinced a change can't break anything. Then it turns out it did. Alongside a battery of tests you trust, you need a way to limit the damage caused by unknown unknowns. That's what I want to explore here. I've put this article in the cynical software category because deliberately testing on users strikes me as pretty cynical. A little cynicism does no harm if both sides come out ahead.

Unknown unknowns

Signing in is one of the few things in any system that can make the whole thing inaccessible if it breaks. GoodData offers several ways to do single sign-on (SSO). A user signs in to their home system, and their identity is passed to GoodData without requiring them to prove it again. The user signs in exactly once. In the older approach, the identity provider sends a signed and encrypted message, a JSON document. It contains the user's login and an expiration timestamp for the GoodData authentication context issued on the basis of the message.

{"email": "user@domain.com","validity": 123456789}

To improve security, we were adding two optional attributes (notBefore and notOnOrAfter) to define when the message itself was valid. The code handling this type of sign-in was originally written in Perl. When it was rewritten in Java, the JSON was parsed by hand. I didn't want to keep hand-parsing JSON as I extended the code. I had a feeling we might run into backward compatibility problems, so I kept the original method as a fallback. The code looked like this.

try{
 return parseStrictly(loginSessionMessage, defaultNotBefore, defaultNotOnOrAfter);
} catch(IOException e) {
 log.info("Cannot parse login message '{}'. Using fallback method for parsing its content.", loginSessionMessage);
 return parseQuirkly(loginSessionMessage, defaultNotBefore, defaultNotOnOrAfter);
}

If the input wasn't valid JSON, Jackson, the JSON library, would throw a subclass of IOException. I deployed this code to production with confidence. The first problem appeared a few hours later, and we had to revert the entire feature. The culprit was this line inside parseStrictly.

final Long validity = (Long) map.get(VALIDITY_FIELD_NAME);

Unfortunately, sometimes someone sent a string in the JSON in place of a number, and the fallback catch block didn't catch ClassCastException. The greatest humiliation was hearing the Perl programmers sitting next to me make sarcastic remarks about how this never would have happened to them in Perl.

This is one of many cases where even the best code review or testing wouldn't have helped. All tests operate on states you know about in advance. A state you haven't accounted for is hard to test. That's what we call an unknown unknown. In this case, the only way we could uncover the problem was by putting the code into production.

A different kind of testing on users

When test coverage is low, developers sometimes quip that they're testing on users. That's obviously a bad approach. Sometimes, though, there is no equally effective way, or no other way at all, to explore an unknown unknown. Every piece of software has at least one undiscovered bug, and tests are only a means of uncovering the rest. Once you accept that, and I don't think we have much choice, you have to try to limit the damage those bugs can do. One technique is to release new features or versions gradually.

The idea is simple. You let users move to the new version in stages. If a bug shows up, it affects only a limited set of users at first. With good system telemetry, you can also detect the problem automatically and roll back to the original version or, for more complicated changes, stop rolling the new version out to additional users. Gradual releases and telemetry with automatic rollback are things we're working on at GoodData and hope to have running soon.

I want to stress that we aren't actually testing on users. The mechanism I described is there to limit damage that can always occur. Any code we release to production must still go through all our quality checks. The mechanism complements testing and adds a layer of protection.

What follows reflects my understanding of the problem at this point and the discussions we've had so far. The implementation may differ later, probably only in the details. I also don't expect us to get everything right on the first try. We'll reach the final state gradually.

Send out the canaries

You can deploy and roll out a new version with a canary release. First deploy the new version alongside the old one and start directing users to it. If all goes to plan, direct everyone to the new version and discard the old one. At GoodData, all HTTP requests pass through one gateway, so we can route each request on arrival to either the old or the new version of a service. We can retain version affinity in cookies, which can pose a problem for programmatic REST clients, or use HTTP headers. For the HTML application, we have a way to bootstrap a particular version. We already use it to support multiple data centers, where different versions may also be running.

Backward compatibility is the hardest part. Internally, GoodData is built as services that communicate over REST interfaces. Each service has its own subcluster and essentially its own lifecycle. Some services are developed quite rapidly and a new version may come out several times a week. Changes released that way therefore have to be backward compatible. An incompatible change must go out during a scheduled downtime window for the whole platform. That's the only way to coordinate incompatible changes across several services.

There are several ways to choose which users receive a release. At GoodData we can roll it out by user or project or by larger groups they belong to. In other words, you slice the release by users or by the projects they use. For example, a new SSO version makes sense to roll out by user. I like a method I call dogfooding, where internal GoodData users get the new version first. In practice, users with an @gooddata.com login or people connecting from our internal addresses would always get the newest version. If no regression appeared, the rollout could continue to more users. Facebook, for example, has a similar model.

A dead canary

The release and any rollback need enough automation. One key part is detecting anomalies that might signal something wrong in the new version. We collect a range of telemetry from production to distinguish ordinary operations from unusual conditions that show up as anomalies. An anomaly can be anything out of the ordinary. The ratio of HTTP statuses, such as internal server errors and client errors, can be a fairly reliable sign of a problem. Taking anomaly detection further, we'd need something that can learn to recognize anomalies from historical traffic data. We'd train an AI system on that history and feed it current production data periodically. When the system detects an anomaly, a person can decide whether to continue the rollout, pause it, or roll back.

Conclusion

However far we take this idea, conventional tests are clearly sufficient only for the states we know about. For unknown states, we need a way to keep any problems confined to a limited and ideally very small group of users. The techniques above help us do that.