Another step toward continuous delivery

I’ve written about continuous delivery in several earlier articles, and in an older one I considered building everything together versus making components independently releasable. Looking back, I think releasable components are an excellent step toward continuous delivery, at least in our environment at GoodData. This article gives you a look at how we develop Java components and shares a few ideas that might help us reach full continuous delivery. For now, we can deliver a new product version about every two weeks.

It all starts with a commit

That heading seems like a natural place to start, but I’ll start with the code organization that has worked for us. Originally we had all our source code in one big pile. At GoodData, we’re building a heterogeneous application. The source for all the services, from the UI client to the ROLAP engine, was organized but still kept together. A special build script for that mix calls the right build tool for each language or platform. For Java, that was Maven. The build produces RPM packages you can use to install the whole product.

In Git, our version control system, we use feature branches to develop new functionality. There’s a Master branch you can merge a feature branch into only after a quality check involving a thorough build and teardown, code review, and a battery of tricky questions. The merge into Master is the bottleneck for the whole process, and code review gets complicated when you change several components. We also have many Java components shared by different services, so we have to be especially careful about changes. When you build everything together, using stable versions of components alongside development versions is practically impossible because the components have different development cycles. We also have mature components we rarely touch. We don’t want them constantly slowing down continuous builds or tying up resources.

Releasable components in practice

For these reasons, we decided to move the Java components gradually out of the big pile and into separate GitHub repositories, then release them as binary artifacts through Maven. We have about thirty GitHub repositories for Java. Each repository has a gatekeeper responsible for the components in it. If I want to change something, I make the change on my own branch and open a pull request. GitHub supports this well, so we don’t have to build anything ourselves. The gatekeeper gets an email with a link to the pull request. They look through the changes in the web interface and have two options. If everything looks good, they accept the pull request and GitHub automatically merges it. If something looks off, they can leave comments on the source or the pull request through the web interface. Those go to the requester by email. The requester addresses the comments, and if there are no further objections, the gatekeeper accepts the pull request. That takes care of code review.

We have post-commit hooks on the GitHub source repositories to run the usual cycle of building, testing and deploying the resulting artifact to a binary Maven repository. The one gap is managing component releases, which the gatekeeper does by hand. To release a stable version, the gatekeeper has to run the Maven Release Plugin manually. It takes care of building the artifact, changing the POM version, assigning the tag and so on. We’d like to automate component releases too, probably by using a special commit message or triggering them manually from the CI server.

Rules

Release notes

Each component has a RELEASENOTES.md file where we record changes in terms people can easily understand, so they don’t have to dig through the commit log. We use Markdown, which GitHub renders in the web interface when you browse a repository.

## 1.1.2
* Added `AbstractMediaTypeAwareFilter` filter provides support for detection  media types from  HTTP headers
* Added `RetainingHttpServletResponse` special `HttpServletResponse` implementation allowing manipulation with the response body and HTTP headers until the flush method is called
* Added `RetainingResponseFilter` this filter wraps `HttpServletResponse` to `RetainingHttpServletResponse`

## 1.1.1
* Added support for altering the Content-Type header (ContentTypeAlteringRequestWrapper)
* Added support for changing the entire request body (BodyAlteringRequestWrapper)

Versioning scheme

We use the usual three-number X.Y.Z version format so the type of change in a given version is apparent at a glance. X denotes the generation of the component. It’s always 1, and I don’t expect that to change unless we decide to overhaul the component completely. Y denotes major changes that can be incompatible with the previous version. I’ll come back to incompatible changes. The last number denotes incremental changes, bug fixes and the like.

If you release components independently, backward compatibility becomes a watchword. You don’t know who’s using which version of your library, where they’re using it, or when they’ll choose or be forced to move to a newer version through transitive dependencies.

Over time we’ve found it’s best to keep changes backward compatible, especially in low-level components such as an HTTP client. Otherwise the whole thing can collapse like a house of cards. Incidentally, that’s one disadvantage of releasable components compared with building everything together. Application components don’t require quite as much caution.

Readme

Most components should come with a description of what they do and how to use them. That makes it easier to get your bearings when using one.

JSON
===========
This repository consists of two parts

* [JSON validation](https://confluence.gooddata.com/confluence/display/analysis/JSON+validation+documentation)
* JSON serialization support - also contains [AbstractJsonObject (de)serialization](https://confluence.gooddata.com/confluence/display/Development/Abstract+JSON+object+serialization)

Continuous delivery

There are a few barriers to continuous delivery, at least for our Java services. Some of our high-level services still can’t be released independently. We do stick to one rule. We don’t use SNAPSHOT dependencies in the stabilization branch, which is forked from Master. That makes the build reproducible and keeps us from breaking it. Once all the services can be released, we’ll be a few steps from our goal. The first barrier is psychological. I call it the sum of all fears. We need enough tests, and I’m deliberately leaving out a code coverage percentage, to let us commit knowing that a problem would be caught.

Another piece is a staging repository for service binaries. Once an artifact is ready in the Maven repository, having passed its unit and integration tests, it needs to be tested with the rest of the system. If those tests pass, we can move it to the staging repository, which supplies artifacts for the final distribution that goes through manual verification.

The last obstacle is that we can’t release or roll back just part of the system. At the moment we can release it only as one big unit, which adds to the cost of the whole release.

Conclusion

The road to continuous delivery is long and rough, and no one hands you a map. All that works is improving gradually and learning from your own mistakes. I hope that in six months I can report how far we’ve come.