How the continuous integration infrastructure evolved at GoodData

Over the past few months I have helped develop our continuous integration (CI) infrastructure at GoodData. I would like to share the design we came up with and eventually put into practice.

Motivation

Two major developments led us to change our CI infrastructure. A little background helps explain them. GoodData had historically been a monolithic application. Rewriting it as a platform as a service brought some changes to the architecture. Around 2009, Perl was the main back-end language and JavaScript was the main front-end language. The version-control system and the CI infrastructure reflected that setup. About 15 developers worked on the whole product. Over time GoodData grew on all fronts, from the size of the codebase to the number of developers.

That growth naturally exposed bottlenecks in team productivity, which we decided to address through changes to the architecture and the organization. We redesigned the architecture in good faith to make it more service-oriented. We assumed teams working on services with little technical debt would be able to build and, especially, deliver new features much faster than every two weeks. We also assumed they could do so independently of services and teams that had more technical debt or that we could not staff adequately for various reasons.

The road to hell is paved with good intentions

The effect on team productivity was the exact opposite of what we had expected. A major reason was that most of our CI infrastructure still reflected a monolithic application. In 2009 we had one Git repository, two programming languages, and 15 developers. By the start of 2013 we had about 20 Git repositories, four times as many developers, and five programming languages.

A short analogy helps explain the situation. Imagine you run an engineering office that designs and builds car prototypes. You draw up a plan, assemble the first version, and test it. It turns out the chassis needs replacing and the body needs a different design. You make another version and test it again. You need a drawing that records exactly which parts make up the car. Since prototype development is iterative, recording that your prototype has four wheels, an exhaust, a body, a steering wheel, an engine, and so on is insufficient. You also need the version of every part.

Without those versions it would be hard to send the car into production because nobody would know what had actually been tested or approved, or what had gone into a given prototype. Each of the more complex parts is designed by a different department. Designing and tuning an internal combustion engine is an order of magnitude harder than designing a turn signal, and sometimes a part needs only a minor change. The departments therefore deliver new versions at different speeds. Prototype development never ends. One version has been refined and goes onto the production line while you work on the next. With no shortage of work, there are prototypes in different states. Version A is on the production line, you are finishing B, C is well under way, and the engine department might already be working on D.

In that analogy, replace each part with a service or component, the prototype with GoodData, versions A, B, C, and D with particular releases, the machinery for making parts with builds, the test driver with automated tests, and sign-off with QA. We are back in software development.

The original infrastructure

The original infrastructure ran in Jenkins and had three main parts, the build pipeline, integration pipeline, and deploy pipeline.

Build pipeline

The build pipeline handles the build and basic quality assurance. It always produces one or more RPM packages. Quality assurance varies with what Jenkins supports for a given technology and what types of tests are available. The absolute minimum is unit tests. If the tests fail, the output does not go to the integration pipeline. Running tests this early is very inexpensive in terms of time.

Integration pipeline

The integration pipeline runs functional tests against an installed version of GoodData. There is no place for mocks here. The services are actually integrated. The functional tests use the REST interface and perform essentially the same operations as any GoodData client, including the GoodData application in the browser.

Deploy pipeline

The deploy pipeline installs GoodData in an environment consisting of a single machine, such as a personal instance, a QA instance, or a Jenkins worker, or in a cluster with the same topology as production.

Note: the gears represent individual Jenkins jobs.

Main shortcomings

With so many changes, the whole system on the main development branch was often broken. Creating a development environment from the current state of that branch was like playing Russian roulette. It took about an hour to create an environment through the deploy pipeline, which made this particularly frustrating.

The integration pipeline did not run automated tests in the same environment developers used. A test passing in a development environment therefore told us little. On top of that, the CI infrastructure for the automated integration tests was unreliable. We saw many false positives and situations where a developer said, “It works in my development environment.” This led to distrust and frustration among developers.

The Jenkins CI server had many jobs that gave some indication of the state of the main or release branch. What was missing was one clear view of the overall state, especially the test results.

These shortcomings made it fairly easy to identify what had to change.

• The development environment must match the environment the CI infrastructure uses to run automated tests. Ideally, it is the very same environment.
• There must be a way to tag the Puppet configuration and the combination of RPM packages that pass the integration pipeline. We should then be able to take that snapshot at any time and create a development or QA environment from it, or deploy it straight to production.
• There must be a way to run integration tests for services that develop at different speeds without creating branches.

The release train

Since we used separate Git repositories, integration had to happen at the level of the binary RPM packages produced by the build pipelines for individual services. That is where the release train came from, a place to collect those packages. Think of a release train as a directory into which a build pipeline copies its binary packages when it finishes. Packages from all services arrive independently. There are always three active release trains at a time.

• The production release train represents the version of GoodData deployed to production. It receives mainly bug fixes that need deploying at the earliest opportunity.
• The stable release train represents the next version of GoodData, which will be deployed within the next 14 days.
• The master release train is the development version of GoodData, due for deployment within about 30 days.

Assembly descriptor

A release train is simply a container representing a release. To see what state it is in and create a snapshot of that state, we need a description that identifies the versions of the binary packages unambiguously. We call that an assembly descriptor. In addition to package versions, it holds test results and other metadata. It is essentially a résumé for the snapshot. It is created when packages are copied to the release train, and it drives both the integration and deploy pipelines.

Assembly descriptor

[Addresses and internal repository identifiers in this example have been replaced with fictional values.]

{
  "artifactGroups": [
    {
      "name": "client",
      "artifacts": ["s3://example-bucket/release-train/r91/example-client-1.2.3.noarch.rpm"]
    },
    
    {
      "name": "webapp",
      "artifacts": ["s3://example-bucket/release-train/r91/example-server-1.2.3.noarch.rpm"]
    },
    ....
  ],
  "creationDate": 1366975570871,
  "id": "s3://example-bucket/release-train/r91/example-assembly.json",
  "labels": [],
  "lastModifiedBy": null,
  "lastModifiedDate": 1366985563243,
  "lifecycleState": "FINISHED",
  "memo": null,
  "puppetVersion": "git@example.invalid:example/puppet.git#0000000000000000000000000000000000000000",
  "tests": [
    {
      "skipCount": 0,"failCount": 0,"totalCount": 0,
      "type": "DEPLOY",
      "result": "PASSED",
      "build": "https://ci.example.invalid/job/example-smoke/1/"
    },
    {
      "skipCount": 0,"failCount": 0,"totalCount": 64,
      "type": "SMOKE",
      "result": "PASSED",
      "build": "https://ci.example.invalid/job/example-functional/2/"
    },
    {
      "skipCount": 0,"failCount": 0,"totalCount": 3645,
      "type": "FUNCTIONAL",
      "result": "PASSED",
      "build": "https://ci.example.invalid/job/example-integration/3/"
    }
  ],
  "version": 1
}

A release train produces about 20 assembly descriptors per day, sometimes more, depending on developer activity. We do not test every descriptor, since a complete run through the integration pipeline takes two and a half hours. Instead we automatically schedule the newest descriptor created in the previous 60 minutes for testing. The integration pipeline takes the descriptor as input and creates a test environment from it. It then runs a battery of smoke and functional tests. The result is written back to the assembly descriptor. The full process is a little more complicated, but the following illustration gives an idea of how it works.

Visibility and transparency

The release train and assembly descriptor are technical means of addressing the main shortcomings of the previous setup. They needed a graphical interface that would allow us to:

• See the state of GoodData development in one place, including how well the tests are passing
• Create a development environment from a tested assembly descriptor

The interface started as a Jenkins view. Over time we added conveniences like labels and the ability to compare two assembly descriptors.

A developer can look at the release train, choose the best available assembly descriptor, and create a development environment from it. With package versions and configuration pinned, the developer can be fairly confident it will work. They can also debug their own code and find regressions by running tests, since the integration pipeline uses the same environment. If they need more information about a descriptor, such as test results, its details link to the individual results.

Architecture

The system has two main parts, back-end jobs and a user interface, and both run in Jenkins. The UI is a conventional Jenkins plugin. The back-end jobs used in the pipelines are ordinary scripts. Nearly all jobs that work with release trains and assembly descriptors are written in Java. Java provides a domain model representing the main entities and other abstractions. There are APIs for operations against the data store and for higher-level operations such as listing all assembly descriptors. The UI plugin uses these APIs as well, which is why we chose Java. Alongside the jobs that participate directly in the descriptor workflow, service jobs such as a garbage collector help keep the system running.

Technology

All data, including RPM packages, assembly descriptors, and other metadata, is stored in Amazon S3. A release train itself is simply a key in a bucket. If you have never used S3, think of that key as a directory in a file system. The assembly descriptor is a plain JSON document. Much of the UI is written in JavaScript using YUI. The integration and deploy pipelines are written in Bash. Since S3 is not a database and displaying assembly descriptors means loading and deserializing them, we cache fairly aggressively with Ehcache.

Conclusion

For practical purposes there were two of us working on the system, myself and Radek Chromý, whom I want to acknowledge here. We spent all of November and December developing the first version to test whether the design was viable. We spent the next two or three months refining it and rolling it out more widely. We resolved the problems with the original CI infrastructure. Tests became substantially more stable because developers could no longer explain away failures by pointing to differences in infrastructure. Creating development environments also became more reliable, as the following chart shows.

In my view, the quality of a system design is tested when it has to be extended. So far, it appears that the changes we are planning can fit into the release train design. We are still hiring if you would like to work on the system and help develop it further.