Self-tests

If your application has more than one possible configuration setting, you probably need a way to check that it’s configured correctly after an install or release and ready to run. That’s especially useful when it runs with different settings in different environments, such as a production cluster, a test environment and a development environment. The checks can cover both the application and its surroundings. The entire GoodData platform cluster is configured with Puppet. Scripts or parts of scripts can fail, so we developed a simple framework for writing tests that verify the configuration of the application and its environment.

What to test

Several areas are worth testing in any web application. We can split them into internal and external checks.

Internal

Configuration properties
 Configuration property tests simply check that the value differs from the default. The defaults are set for the local environment. Production, for instance, shouldn’t use the same password as a local environment, so we check the value of the setting that supplies it. We usually check for differences from the defaults or for acceptable ranges. The number of threads handling HTTP requests, for example, should exceed 100.
 Components and their settings
 These tests check the settings of individual components, especially ones that behave differently in different environments. A component can work with its default settings, yet we need to know that it will behave differently in production. Take our single sign-on (SSO) component or the cache. We check that SSO has a keystore and can access the keys. For the cache, we check that replication works by making sure its configured hosts and ports are accessible.

External

HTTP and messaging endpoints
 Every application uses external services, and it’s worth checking they’re available. A simple request can do that. When the nature of a service makes this impossible, for example if delivery can’t be verified, we check that the host and port are reachable.
 Databases
 We check the database, MongoDB in our case, by listing the collections it makes available. A more advanced test can compare the application version with the database version to avoid problems such as inconsistent data.
 Runtime environment
 These tests check the settings for the JVM and Tomcat. We verify that the JVM has the minimum heap and PermGen sizes we expect, that we’re really using the version of Tomcat we tested with, whether a security manager is configured, and so on.

How to test

One of our design criteria was to separate each test from the way its results would be presented. We also wanted to make it easy to add tests without touching the system core to register them. We addressed the first requirement with the standard logging object org.slf4j.Logger and implemented our own result processing behind it.

Interface

/**
 * Interface representing self test of java components of GDC platform.
 */
public interface ISelfTest {

public static final String LOGGER_PREFIX = "SelfTest: ";

/**
   * Tests desired functionality and logs results into logger property (of type {@link Logger}).
   * The pattern is that problems are logged as ERROR, ok messages as INFO.
   */
   public void test(Logger logger);

}

A test that checks the JVM type

Note that some parts of AWT and Swing differed considerably from one OpenJDK fork to another. We needed this test to be sure the right JVM was in use.

/**
 * {@link ISelfTest} checking whether the active JVM comes from Oracle or Sun.
 */
public class SunJVMSelfTest extends SelfTest {

@Override
  public void test(Logger result) {
    String vendor = System.getProperty("java.vm.vendor");
    String version = System.getProperty("java.version");

if (vendor != null && (vendor.toLowerCase().contains("oracle") 
      || vendor.toLowerCase().contains("sun"))) {
      result.info(LOGGER_PREFIX + "Java vendor='{}', version='{}'.", vendor, version);
    } else {
      result.error(LOGGER_PREFIX + "Java vendor='{}', version='{}'.", vendor, version);
    }
  }
}

To save each component from registering its tests in the core, we used the inversion-of-control framework’s ability to return every component of a given type. See Automatically collecting objects of a given type with Spring. We load the components dynamically, so the existing core configuration needs no changes.

Running tests and presenting results

The way we run the tests is independent of the tests themselves. In theory we can start them through JMX or a RabbitMQ message, but the mechanism we actually implemented uses HTTP. A dedicated servlet runs the tests. We return the results as JSON in the HTTP response and log them too. That makes them available online for analysis if needed.

Sample output

{
   "selfTestResult":{
      "testsCount":13,
      "errorsCount":3,
      "warningsCount":0,
      "infosCount":16,
      "errors":[
         "c.g.s.t.a.ApplicationCodeSelfTest: SelfTest: Master password is not changed.",
         "c.g.s.t.a.ApplicationCodeSelfTest: SelfTest: Pool size is set to default value.",
         "c.g.s.t.jvm.MemorySelfTest: SelfTest: 'Maximal memory (-Xmx)' is below desired bounds (current='516947968' desired='1073741824')."
      ],
      "warnings":[

],
      "infos":[
         "c.g.s.t.jvm.MemorySelfTest: SelfTest: 'Init memory (-Xms)' is within desired bounds (current='536870912' desired='268435456').",
         "c.g.s.t.jvm.MemorySelfTest: SelfTest: 'PermGen space' is within desired bounds (current='1073741824' desired='268435456').",
         "c.g.s.t.jvm.MemorySelfTest: SelfTest: Used memory '337501408'.",
         "c.g.s.t.jvm.SunJVMSelfTest: SelfTest: Java vendor='Oracle Corporation', version='1.7.0_02'.",
         "c.g.s.t.jvm.TmpPathSelfTest: SelfTest: Temp path='/mnt/tmp'.",
         "c.g.s.t.s.ApacheSelfTest: SelfTest: Project templates are accessible on uri='/projectTemplates'.",
         "c.g.s.t.s.C3SelfTest: SelfTest: C3 service is listening on host='localhost' and port='667'.",
         "c.g.s.t.s.C3SelfTest: SelfTest: C3 service FoodMart demo is accessible on uri='/gdc/c3/project/FoodMartDemo'.",
         "c.g.s.t.s.ConnectorsSelfTest: SelfTest: Connectors for FoodMart demo are accessible on uri='/gdc/projects/FoodMartDemo/connectors'.",
         "c.g.s.t.s.JBossCacheSelfTest: SelfTest: JBossCache works ok.",
         "c.g.s.t.s.MetadataSelfTest: SelfTest: Metadata for FoodMart demo are accessible on uri='/gdc/md/FoodMartDemo'.",
         "c.g.s.t.s.MongoSelfTest: SelfTest: Connection to mongo host='127.0.0.1' port='27017' ok. Available databases='[gdc, local]'.",
         "c.g.s.t.s.MongoSelfTest: SelfTest: Connection to mongo host='127.0.0.1' port='27017' database='gdc' ok. Available collections='[xxxx]'.",
         "c.g.s.t.s.RabbitSelfTest: SelfTest: Rabbit MQ service is listening (host='localhost', port='5672').",
         "c.g.s.t.s.StageDirectorySelfTest: SelfTest: Stage directory='/var/gdc/users' is accessible and writeable.",
         "c.g.s.t.tomcat.TomcatSelfTest: SelfTest: Tomcat major version is ok. ServerInfo='Apache Tomcat/7.0.23'."
      ],
      "includedTests":[
         "ApplicationCodeSelfTest",
         "MemorySelfTest",
         "SunJVMSelfTest",
         "TmpPathSelfTest",
         "ApacheSelfTest",
         "C3SelfTest",
         "ConnectorsSelfTest",
         "JBossCacheSelfTest",
         "MetadataSelfTest",
         "MongoSelfTest",
         "RabbitSelfTest",
         "StageDirectorySelfTest",
         "TomcatSelfTest"
      ]
   }
}

Further development

There are two ways to extend this. One is to write new tests and expand the existing ones, which is a never-ending battle. The other is more comprehensive testing of cluster deployment. I’m thinking of orchestrating tests with other services, for instance, to verify that our services and the cluster as a whole work. That’s a slightly different kind of task.

Self-tests help us find problems that can crop up during a release. Most platform components still release every two weeks, but some are released several times a week. The long-term goal is continuous delivery, and I can’t imagine doing that without a tool like this.