The (not entirely) futile fight against technical debt

Our solution is technically complex, and it would be even if we weren't shooting ourselves in the foot. If you're building a distributed system, it's going to be complex. Our complex system also has the added spice of several languages and types of middleware. That makes it important to tackle technical debt so skeletons don't keep falling out of the closet. At GoodData we've had mixed success. Here are a few observations and techniques we've tried.

Everybody change places

For a long time we split the whole system horizontally, with each team looking after its own layer or service. We didn't do particularly well at tackling technical debt that way, and that wasn't the only problem. We also struggled to build features for ordinary product requirements, especially when teams had to work together. We made a big-bang change and reorganized the teams vertically. Before that we had a team of Erlang developers, a team of Java developers, one for JavaScript and one for Perl. Now, figuratively speaking, each team has an Erlang developer, a Java developer, a Perl developer and a JavaScript developer.

That setup by its nature doesn't do much for paying down technical debt. Or more precisely, I don't see any significant difference. I'd say we confirmed that moving people from one side to another makes no difference to technical debt, even though that wasn't the main goal of the reorganization.

Admit you have a problem and quantify it

It's like an Alcoholics Anonymous meeting, or so I imagine, since I've never been to one. First you have to admit there's a problem. We made a list describing each area and set priorities. It hangs on the kitchen door where nobody can miss it. Make it visible. It matters that it's something concrete and quantifiable rather than an abstraction.

Agree to do something about the debt

It's important to agree with management that dealing with the debt is in everyone's interest. Management can have a tendency to play these problems down, so it's important to explain, explain and explain again, as often as it takes. Technical debt won't disappear with a wave of a magic wand. Managers' allergy to any mention of technical debt comes from engineers' habit of using it as an excuse at every opportunity. That's exactly why you have to agree that you want to tackle it.

Make the work a priority

At the start of every quarter we make a list of priorities to focus on. Whenever a specific item from the debt list didn't make it onto the priorities, nothing happened and the debt remained. It isn't enough to make tackling technical debt a priority in general. Pick a specific item from the list and give it the same priority as a product requirement. In Scrum we tried reserving some team capacity, but that didn't give brilliant results either. What seemed to work best was dedicating a whole sprint to one specific area of technical debt.

Ownership

Technical debt often surfaces in places owned by everyone and by no one. It's like a courtesan who sleeps with half a football team. When she gets pregnant, nobody claims responsibility, because it could be anyone's. It makes no difference whether it's a part of the codebase or a shared service. Clear ownership makes planning and allocating resources easier. If you can't name an owner, you have to break the thing up somehow. If that's impossible, it usually points to some unhealthy dependencies. Untangling them often turns out to be where all the difficulty lies.

Patience

Arm yourself with patience. Paying down technical debt takes a long time. It becomes tilting at windmills when ownership is unclear, priorities clash and you also have fires to put out. In that case, go back to the start of the list above.