Velvon retrospective II: Organizing development

In this post I’ll look at feature development within one team, from the idea to the rollout. From the start, we wanted a team and a product owner responsible for designing, building, rolling out and maintaining what they developed. The product owner was to own a feature from the initial idea through pilot validation, from a minimum testable product to a lovable product, all the way to rollout through feature toggles. A big challenge was developing and architecting the mobile app, including the server API it needed.

Image could not be loaded. Organization chart for the Velvon development teams

We had what we called feature teams. Each looked after one business domain within the bank, such as payments, lending products or customer onboarding. These teams could build the entire backend. We also had a mobile team responsible for the iOS and Android banking apps. It was shared by all feature teams that lacked mobile developers. The feature teams were responsible for providing APIs oriented around use cases, as opposed to the low-level technical APIs used between microservices. See the Backend for Frontend design pattern, which I’ll call B4F from here on.

Distributing B4F across feature teams meant that the only work outside a team’s planning capacity and decision-making authority was development of the mobile app itself. We weren’t considering internet banking in 2018. The disadvantage, at least in our implementation, was that each team offered a different B4F API. Every village had its own dog, as the saying goes. Some used REST, others RPC or GraphQL. The mobile developers weren’t happy about having to use different APIs depending on what they were working on. With that zoo of API approaches, it was hard to create even a modest shared, reusable abstraction on which to build a reactive UI.

Image could not be loaded. Distributed B4F, with each team providing its own APIs

So I decided we would move to a centralized B4F service. The idea was to have a consistent approach at the API level that would suit mobile development as well as possible. The mobile and B4F teams chose GraphQL, which proved to be the right choice, especially after we decided in mid-2019 to start developing internet banking. It could take full advantage of the existing GraphQL API.

We all realized that we had traded a technical problem for an organizational and planning problem. Suddenly both mobile development and B4F were outside the feature teams’ planning capacity and decision-making authority. We wanted to solve this by first finding the holy grail of doing mobile APIs well and building the infrastructure around B4F. Once we had that, we would let feature teams contribute to B4F through internal open source along the lines of Spotify’s model.

Image could not be loaded. Team services behind a single GraphQL interface in the centralized B4F

The future looked rosy, at least until we started thinking about how the app would behave in real time. The original architecture used background push notifications to tell the app that something had happened on the backend. The app would then fetch the data via the API. It was slow and unreliable.

We started considering WebSockets, but at the time there was an issue with them in Azure’s infrastructure. David Vávra, our lead Android developer at the time, suggested moving the mobile architecture in a reactive direction. We tried different services and eventually chose Google Firestore, which I mentioned in the first installment.

Google Firestore was the right choice, but there were tradeoffs. It became increasingly clear that contributions to B4F from the feature teams would be difficult. An architecture based on simple HTTP request and response, familiar to every developer, became a CQRS architecture. Feature teams could make simple changes, but the B4F team handled anything more complicated.

It was complicated organizationally and from a planning perspective. Feature team A and product owner X had to reach agreement with both the mobile team and the B4F team. A single story would be refined at least three times, once in the feature team, once in B4F and once more in the mobile team. Because the B4F and mobile teams (iOS and Android) were shared and therefore bottlenecks, we had to plan and prioritize centrally. Mobile and B4F capacity was rationed every two weeks. Teams that didn’t get any could at best prepare the backend, with all the drawbacks of working in pieces, building the backend first and only then the frontend.

We wanted the product owner to have complete control over the feature: backend, mobile (iOS and Android), internet banking and B4F. We tried this with the team onboarding new customers and the team working on a lending product. We simply added developers with the relevant skills, iOS, Android and B4F as needed. We wanted to continue the transformation. Where was the problem?

After a while, each team had found its rhythm and developed its own modus operandi (management, QA, refinement, estimation, preparing and writing stories in JIRA, retrospectives) and its own working culture. Moving developers around in any way would have been delicate. We considered a core mobile team and a guild of mobile developers. All mobile developers would belong to the guild so information could still flow between the core and feature teams. They would join feature teams for a limited time, then return. We wanted to pilot this with other teams, but we never got to do it.

I still wonder how we could have done it better. Could several teams have developed and tested the mobile apps, internet banking and the necessary server API (GraphQL + Google Firestore) in a distributed way, coordinating only the release and rollout in the app store? That raises several questions.

• How do you manage the app architecture and introduce conventions and rules to guide development? For example, it probably wouldn’t be wise for one team to use MVVM and another to use MVP.
• Feature teams didn’t have the same amount of mobile work every sprint. We would need a backlog of technical work, or developers would need to help other teams.
• Headcount would rise considerably (iOS, Android, web and backup coverage), making feature teams larger and Scrum routines more complex.
• Who handles on-call support for what is essentially a shared codebase of B4F microservices providing the server API (GraphQL + Google Firestore)?

I admit it’s a story without a happy ending. If you’re interested in engineering organization, I recommend these articles by Nick Tune.

• Product-aligned vs Capability-aligned Organisation Design
• Should We Create a Shared Service? A Decision-making Checklist
• Organisational Dysfunctions Mirrored as Architectural Complexity

And of course https://labs.spotify.com/2014/03/27/spotify-engineering-culture-part-1/