Sunday reflections on software development

Plenty of people think about how to make software development more efficient. I worked with a methodology that was classic waterfall with milestones every six weeks. It began with gathering requirements, followed by analysis, design, implementation and finally verification and testing. In practice a different team did each job. Functional architects gathered requirements, technical architects did the analysis, developers designed and implemented the software, and the quality department handled verification and testing. The system worked, but it could have worked much better and more efficiently.

The first problem is the fairly rigid division of responsibilities. It may seem logical and proper, but in practice it defeats its purpose. Each group has a clear start and finish, and another group picks up its work. This brings an absurd amount of duplication in sharing information. If person A at the start of the chain comes up with something or changes it, that information has to reach all the groups downstream, whenever the change happens.

To avoid repeating information at every handoff, we create documents. We end up with a whole series of documents in different revisions describing exactly what the author intended. We have one place to keep the information, but some people have to spend time writing it down and others have to spend time taking it in.

In general I think dividing development into silos simply fails for social reasons. One group comes up with something and a month later another group discovers it can't be done. Unnecessary friction follows when the architect asks for the Taj Mahal and the developers can deliver, at best, a Canadian lumberjack's cabin or, at worst, a woodshed.

The second problem is that analysis is fixed at the start of the process so we can estimate the effort and let other groups get to work. That has a fairly serious shortcoming. We can't always see everything under the hood. More often we see only the part we know well and miss the impact on everything connected to it. The analysis therefore rests on shaky ground from the outset. Try estimating the time needed under those conditions.

We've covered some of the drawbacks. How do we refactor the process so it works for everyone involved?

The first problem to tackle is how to remove barriers to sharing information.

Writing ten-page tracts for functional requirements makes no sense. The clearest approach is to sketch how the application's user interface should behave and add a brief description. If you can't do that, the specification is probably too complicated. When it looks that way at the user-interface level, the underlying request may be perfectly simple and you may have misunderstood what the user wanted. If users want a scalpel, give them a scalpel. They have no need for a Swiss Army knife with a screwdriver, corkscrew and file.

Write an analysis only when you truly need one, and avoid using it to assign work to someone else or estimate how long it will take. Put off the important decisions that tend to go into analyses until the last responsible moment (see

Lean software development), when you have all the information you need.

An agile software development approach can move the building of options earlier for customers, thus delaying certain crucial decisions until customers have realized their needs better. This also allows later adaptation to changes and the prevention of costly earlier technology-bounded decisions.

A software developer's job goes beyond writing essays or wading through them. The most efficient way to exchange information is to talk directly and involve every member of the team in every part of the development cycle. Strictly dividing the work just puts up barriers. It also leads people to specialize in one part of the solution. Friction follows when the people working on the user interface feel shut out of the system's core, and the people working on the core have no idea what the interface team needs. The same caste system causes problems when we divide people into developers and architects.

It may be anarchy,

and it may be cowboy coding, but I think everyone should do everything. It's the best way to share knowledge across the team and it gets people talking more. People enjoy it more, of course. There is no rigid distinction between architects and developers. The team makes important decisions together. Developers respected by the rest of the team serve as “architects” and have the authority to arbitrate a disagreement, say over using library X or Y.

An aside about the manager's role. The manager provides services to the rest of the team. They take meeting minutes, get hardware and software, keep track of deadlines and remind people of them, and coordinate with other teams. Essentially, they shield the team from bureaucratic constraints. The manager never makes technical decisions, although their opinion is welcome and respected.

Sometimes I feel that we put too much stock in development processes and methodologies. From what I've written you might think I've refactored waterfall into Scrum, for example. I admit I don't know. There are plenty of agile methodologies. Sometimes we're playing with words and end up with the same result. I believe in two things in software development.

• Communication, communication and more communication in place of bureaucracy that tries to stand in for it
• Strong individuals at the core of small teams, with every member taking part in every activity

It hardly matters whether you call it Scrum, iterative and incremental development, or lean software development, or try to find those methods in it.