Monitoring Java applications in the cloud

One lesson the cloud teaches you when you host your own services is that the old approaches that worked for years without a second thought stop applying. At GoodData we use a DevOps model, which means developers take their work all the way to production. We don't have a bunch of guys in a server room with nothing else to do but watch what's happening. Our production cluster runs N JVM instances, and we can start and stop them dynamically according to expected or known load. We also have dozens of instances on development and test servers and on developers' own instances. We need monitoring for all those running JVMs and the applications inside them.

I've always been excited about how many tools are available in the Java ecosystem. Unfortunately, I hadn't realized how much my view was limited to the needs of an everyday developer. We have simple profilers and monitoring tools like VisualVM and full-featured profilers, but they're meant for use during development.

At GoodData we have two types of deployment environments for the Java artifacts or components we host. One type runs in a servlet container. The other, which is central for us, is a standalone process. We looked at what we could do with NewRelic, JavaMelody and Hyperic and I was very disappointed with the results.

Requirements

For administration I want a single dashboard at one address where I can find the instance I'm interested in and look at its details. I want to add statistics I care about. In addition to looking at them, I need to be able to define them somehow. For example, I may want the number of certain API requests, how much data moves through particular parts of the system or how much time it spends there. In other words, I need to be able to customize the monitoring.

The basic JVM metrics we're interested in are:

• maximum heap size and heap usage over time
• CPU usage over time
• how much memory the garbage collector reclaims and how often it runs over time
• number of classes over time
• thread states over time

Taking the tools in order, New Relic and JavaMelody are heavily geared toward monitoring web applications, and they provide a lot of monitoring I won't use at all. New Relic can monitor background processes, but the pricing isn't very friendly given how many instances we need to monitor. Hyperic looks more general-purpose and lets you create your own metrics from JMX beans. On the other hand, it's hard to find your way around, and Java looks like an afterthought.

Conclusion

I've decided that unless a tool that has managed to escape my notice miraculously turns up, we'll just write the monitoring ourselves. No, don't throw up your hands. I'm quite serious. We already use Splunk, which lets us collect information from logs across every instance, query it and build tabular reports and charts from the results. That's the administration I'm after. We'll write a simple agent in Scala or Java that latches onto the JVM, slurps up its data and writes it to logs in a format we can easily mine in Splunk. That'll give us live monitoring. It'll also be easy to write monitoring probes of our own. To track longer-term trends, we'll build a connector to send the data to GoodData.

Note: At the next CZJUG meeting, on 28 March 2011, I'll talk about how developing in the cloud and working on a product sold as a hosted service, or SaaS (Software as a Service), changed my perspective and the habits I'd picked up in enterprise Java.