The Road to Continuous Delivery I: Mainline Development

I’d like to move one of our components from feature-branch development, with one branch per feature, to mainline development. On a feature branch, a developer tinkers with changes on a branch of their own, merges changes from mainline occasionally, and merges the code back into mainline when they think everything is ready and quality assurance has given them the green light. In mainline development, with a few exceptions, there are no branches. Developers make all their changes directly on mainline, at least once a day. Both approaches have advantages and drawbacks.

Feature branches

The main advantage is that the code can change many times before the developer is happy with it. This also works very well with code review. Nothing goes into mainline without at least one other pair of eyes on it. The disadvantage is that integration with mainline happens only sporadically. You can be sure that there are no integration problems only once all the integration tests have run against the rest of the system.

Mainline

The main advantage is that any problems show up as quickly as possible. Developers don’t sit on changes in their own branches only to discover integration problems at the code drop, when everyone merges into mainline. The drawback is that it demands much more of developers. They have to think through changes and make them in smaller steps, committing at least once a day. Code review is very difficult because you don’t have a complete picture of how the end result will look.

At most three developers work on this component at a time, at least for now, so we don’t run into problems merging changes into mainline. That’s certainly one of the main arguments for continuing to use feature branches. The argument against them is that this component plays a critical role in the platform, and bugs can have serious consequences. On top of that, a change only really takes effect after the code drop for the other components that depend on it. That means at least a week spent stabilizing things and sorting out integration problems. Moving to mainline would eliminate the problems caused by delayed integration. The tools around the Java platform should also let us put the necessary practices in place very quickly.

Mainline development means clearing away several technical and, let’s say, social obstacles. Let’s start with the social ones, because they determine the technical ones. Developers worry about the following consequences.

• On a feature branch I can start without knowing exactly how the result will look. I have a rough idea where I want to go, but I don’t yet know the best way to get there. I make a lot of changes before I’m satisfied. I wouldn’t want those changes in mainline.
• I’m not used to planning changes so that I can integrate them into mainline.
• I don’t see how anyone will review my code. We’ll keep interrupting each other, and reviewers won’t be able to judge the changes in the context of the entire feature.

At first glance, that looks like a fairly convincing list of reasons to call off mainline development. And it is, if we leave our idea of what the main branch is for untouched. Mainline calls for a different approach altogether. I’ll try to explain it by describing what changes about the main branch.

Before
The main branch contains code that has passed review and shouldn’t change substantially unless we need to change its behavior or fix a bug. Changes reflect the final state of a feature.
After
The main branch contains code that we know compiles and has passed automated quality checks such as tests, test coverage measurement and dependency analysis. Changes on mainline may be more extensive. The interfaces between the parts stay as stable as possible, while implementations change frequently.

There’s a big question mark over code review. My first thought was to make it voluntary. Another option was pull requests: you make a change, create a pull request and have someone review and merge it. Hybrids are possible too, such as merging bug fixes automatically and using pull requests for new features. None of these options particularly appeals to me, because they once again delay merging into mainline and therefore integration with the rest of the system.

I’d defer code review until the entire feature is in mainline. Changes requested during review can be made incrementally on mainline. The feature stays hidden until we’re all happy with its quality. With a bug fix, we should be able to back out the change easily by reverting the commit if something looks wrong, using the changelog as a guide.

When I think about it, quality assurance in mainline development happens after the fact. Once you make the change, you put it through a process meant to ensure its quality. With feature branches we try to ensure quality up front, putting changes through that process first. In the end, both approaches make it very easy to back out a change.

Technical prerequisites for mainline development

Developing on mainline requires quick, reliable feedback on the changes we make. That’s where the commit stage comes in. It’s the part of the build every change has to pass, and it should produce the binary artifacts used in later stages, such as integration tests and manual tests. If a change fails, it should be fixed or reverted as quickly as possible. At a minimum, the commit stage should compile the code, run unit tests and produce binary artifacts. We can add things like Maven module dependency analysis or test coverage measurement.

Conclusion

I’ve tried to outline the issues around mainline development, a small step on the road to continuous delivery. I’d like to share where this effort goes next in a series of follow-up articles.