Reuse versus agility

I originally wanted to write about the importance of making your work visible and the effect of putting up a TV with an operations dashboard for our production system. Then a few things happened that reminded me of a subject I’ve been thinking about for some time. It has to do with how we approach reuse as engineers and programmers. Code reuse is only part of it. I’m more interested in how reuse relates to agility and how, beyond a certain organization size, it can work against it. I’ll illustrate that with the five different frameworks we use to test our REST API.

It sounds alarming when I say we have five frameworks in various stages of use for testing our public and internal REST APIs. I should explain how we got here. Much of the codebase was originally written in Perl, so the test framework was in Perl too. I don’t know if there was an open-source framework at the time and it doesn’t matter now. We’d lived with ours since time immemorial. Once we brought in other languages, Java and Erlang, a quiet revolt began against using our own test framework, especially one in Perl. There are few certainties in programming, but here’s an unshakable one. Nobody wants to use a language other than their own. A Perl programmer would sooner give up a little finger than write something in Java, and vice versa. It’s a matter of religion, and there’s no getting around it. You can order people to do it, but all you’ll get is “sabotage” and holy wars. In our case that reluctance led to poor test coverage.

Paradoxically, the company’s growth helped. The number of developers multiplied. Alongside organizational changes, the application architecture had to move toward services. In the new setup, teams had autonomy and full responsibility for how they ensured the quality of their services. In effect, the requirement to use one technology for all testing gave way to the practical idea of using the right tool for the job. Different teams began developing their own testing strategies. We ended up with several test frameworks, but also significantly better automated test coverage and even more willingness to write tests.

Why not coordinate it all, build one framework that works for everyone and reuse it? Aside from the dogmatic rejection of other languages I mentioned, there are practical reasons. How long would it take to build, and how would we make sure it suits everyone? Let me ask a question in return. Is it better to have five test frameworks with some duplicate functionality and 50 percent test coverage, or spend six months designing and building a universal framework for a 5 percent increase in coverage? The first option strikes me as far better, even if it isn’t ideal.

In my experience, infrastructure projects designed from an ivory tower don’t work. Perhaps the teams don’t know how to work together in an agile way and focus instead on detailed but incomplete specifications. The result tends to frustrate everyone involved. On the other hand, just because I’ve never seen a black swan doesn’t mean there aren’t any.

Software development thrives under conditions that mirror a free market. Give people freedom, with as few rules constraining them as possible, and give them choices, with the responsibility that comes with making decisions. The solutions they create may compete with each other, but sooner or later the best ones will rise to the top. That’s why I think our five frameworks will gradually settle into two main ones that the rest of the company will adopt. The great advantage is that this happens from the bottom up, through natural selection by the people who actually use the tools. That’s a feature of agile development, too. It happens without Big Up Front Design. Agility should take precedence over the premature optimization of reusable one-size-fits-all solutions, even at the cost of some inefficiency.