Notes on Release It!

I was thrilled by Release It!. Although it came out almost seven years ago, in 2007, it’s full of useful advice and observations on designing and deploying applications for production. The subtitle puts it more precisely: Design and Deploy Production-Ready Software. In my short Goodreads review I wrote that I wished I’d read it three years earlier. It would have saved me a few sleepless nights putting out fires in production.

The book starts from a simple premise.

Enterprise software must be cynical. Cynical software expect bad things to happen and is never surprised when they do. Cynical software doesn’t even trust itself, so it puts up internal barriers to protect itself from failures. It refuses to get too intimate with other systems, because it could get hurt.

Today’s applications, especially web applications, are rarely monolithic in their dependencies. They’re made up of several services, such as payments and email. The familiar term for an architecture made up of services is Service-Oriented Architecture (SOA), a name that unfortunately became rather overused. Whether the services are internal or external, every interaction with one carries a risk.

The snowball effect and a spiderweb crack

Imagine you’re driving and a pebble flies up from an oncoming car and chips your windshield. A spiderweb forms. Cracks spread from the center in every direction. If you’re lucky, you can still drive to a repair shop. Otherwise the entire windshield shatters and the best you can do is pull over and wait for a tow.

There’s one certainty in the life of software: Bugs will happen. They cannot be eliminated, so they must be survived instead. Since applications are now assembled from services, a bug in one can spread to another very quickly and set off a chain reaction, a snowball effect.

The biggest issues stat with something very small. A tiny programming error starts the snowball rolling downhill. As it gains momentum, the scale of the problem keeps getting bigger and bigger.

The spread of failures from one service to another within a system is described by failure modes. The rule is that a failure in one part of a system makes a failure in another part more likely. System complexity amplifies this, giving a failure many ways to spread.

The original trigger and the way the crack spreads to the rest of the system, together with the result of the damage, are collectively called a failure mode.

No matter what, your system will have variety of failure modes. Denying the inevitability of failures robs you of your power to control and contain them.

Once you accept that failures will happen, you have ability to design your system’s reaction to specific failures

Stability

Stability is a system’s ability to withstand failures or errors in its individual parts. Failures start with triggers that overload parts of the system. Imagine your marketing department puts out a discount coupon for an item in short supply, or you make the front page of a site like Slashdot. The sudden horde of users rushing to your application overloads parts of the system, which may then fail. That failure can spread in all sorts of directions.

The book describes antipatterns that make a system less stable and patterns that make it more stable.

Circuit breaker

As an example of a pattern that improves stability, I’ve chosen the circuit breaker. It’s the pattern behind the library Hystrix (which is worth a look). An integration point is where your application connects to another service. Failures in that service either gather speed there and cascade through the system or stop there. A circuit breaker is a component used at integration points to keep failures from spreading.

A circuit breaker is essentially a proxy with three states that govern calls to the service.

• Closed. Requests go through. A failure increments the failure counter, and reaching a threshold switches the circuit to open.
• Open. Calls aren’t made. The caller immediately finds out that the call can’t go through, so it can fail fast. When the timeout expires, the circuit switches to a trial state.
• Trial, or half-open. Requests are let through. If they succeed, the failure counter is reset and the circuit closes. If they fail, the circuit opens again.

For a circuit breaker to work properly, you need an isolated thread pool and, of course, timeouts so that threads don’t remain blocked.

Deployment and transparency

Transparency is one of the basic things a system needs when it goes into production. Developers often believe their work ends once the code passes review or it’s formally handed over to quality assurance. Nothing could be further from reality.

The true birth of a system comes not on the day that design and development begins or even when the project is conceived, but on the day it launches into production.

System transparency is primarily about visibility, on several levels. The system’s overall status should give anyone an idea of how healthy it is right now. The book describes a dashboard with a simple red, amber and green traffic light to show the current state. It monitors and evaluates the following things.

• Whether expected application events happened, such as backups and garbage collection.
• Whether unexpected events happened, such as a server crash.
• The state of application components and services, such as the state of a circuit breaker.
• Whether all monitored parameters are within normal limits.

There should also be a dashboard or tool for seeing more detail, farther down in the system. That typically means looking directly at monitoring sources such as logs. The overall status gives you a picture of system health. This more detailed view helps you troubleshoot current problems. For that we use Splunk.

Without transparency the system will drift into decay functioning a bit worse with each release.

Systems can mature well if and only if they have some degree of transparency.

Anomalies and events can remain forever mysterious or they can be valuable lessons learned. Transparency makes difference between a system that improves over time in production and one that stagnates or decays.

Monitoring tools and technology

There are two approaches to monitoring, known as white-box and black-box. The difference is whether they operate inside or outside the system being monitored.

• White-box monitoring runs inside the system and typically means logging.
• Black-box monitoring runs outside the system. It is usually a process that probes the system.

You may be wondering why logs aren’t enough. If a monitored system hangs, it usually stops producing logs too. Black-box monitoring can at least give you information that may help you find out what’s happening, such as CPU and RAM usage, I/O, network activity, and JVM garbage collector behavior.

The main tool for monitoring a JVM is JMX. If you’ve never used it or don’t know much about it, I’d suggest taking a look. It does management as well as monitoring. Exposing individual application components through JMX allows you to manage them dynamically. We use it at runtime to change log levels, run health checks, and adjust thread pool sizes. Incidentally, being able to restart individual application components and services instead of entire servers is one of the requirements of recovery-oriented computing.

Here are some things worth monitoring and exposing through JMX or whatever other technology you use.

• Integration point health and status, including error counts.
• Traffic indicators, including request counts, duration and data size.
• Resource pool status, including active and idle thread counts, maximum threads and time spent waiting for a thread.
• Database connection pool status, including active and idle connection counts, maximum connections and time spent waiting for a connection.
• Cache status, including the hit-to-miss ratio, how full individual regions are, and the number and size of objects.

Recap

Of course I can’t fit all the wisdom in the book into one article. It goes into considerable detail on capacity and capacity planning, the SLA for an application on its own, and how external services affect that SLA. I’d certainly recommend reading it. A few parts are geared toward Java, though, such as tuning and garbage collector behavior. Don’t expect guidance on running and deploying applications in the cloud. It struck me as pioneering for its time in the way it emphasized some of the ideas that DevOps is built on.

Don’t avoid one-time development expenses at the cost of recurring operational expenses.

You early decisions make the biggest impact on the eventual shape of your system. The earliest decision you make can be the hardest one to reverse later.

I’m also including a mind map of my notes.