dagblog.
← Archive

Velvon retrospective I: Technology

Roman Pichlík ·

Every failure deserves a postmortem. We tried to make that a habit in Velvon engineering, and I intend to keep it up even though Velvon was shut down on 13 February 2020 and I found myself out of work for the first time in nearly twenty years. I plan to stay that way for a little while. I’ll be writing a few retrospectives about Velvon here on Dagblog. I’d like to start by looking back at what we managed to build.

Velvon was meant to be a modern digital bank for Germany. Development began almost two years ago. I joined after the first five months to help bring over some of the ideas we had successfully implemented at Zonky. Both Zonky and Velvon belonged to the same group controlled by Petr Kellner.

The original team had already gone through several iterations of building the bank. In 2018 it started over from scratch using modern tools and practices: microservices, of which we had 93 by the end, Kotlin, Docker for application deployment, Kubernetes, Azure and DevOps, to name a few. We built an observability platform for monitoring and operating the bank using Splunk, Prometheus, Grafana, VictoriaMetrics and PagerDuty. Our deployment model was fairly straightforward. Each change went through just one environment with automated tests before it reached production. It took 40 minutes from merging into master for the change to appear in production.

On paper and in practice, we had the basic processes to make changes, keep the bank running, run load tests and conduct chaos engineering experiments. Those experiments helped us make individual parts of the system more robust. We had mobile banking for iOS and Android, both reactive, and responsive online banking where the Back and Forward buttons worked (sic!). It all worked well enough to simulate a live bank, including incoming payments and card transactions.

When our CEO, Erich Čomor, told us on Thursday, 13 February 2020, that it was over, hardly anyone could believe it. We had come so close. I’d like to pick out a few things, with the benefits and drawbacks as I saw them. This is a subjective account. Some of my former colleagues may see things differently because I was looking at them from the distance of a head of engineering in an organization with more than 50 developers. I no longer saw every detail, and most of my hands-on work was limited to problems of overall design.

Microservices

Because we started from scratch, we could avoid familiar anti-patterns from migrations away from a monolith, such as sharing a database. Docker and Kubernetes helped standardize deployment, which was a clear win. Some of our services were quite fine-grained and the architecture seemed fairly complex to me, though that may just take getting used to. The following practices worked well for us.

  • We documented service contracts in Swagger. Each build published the contract as part of the service’s documentation.
  • We checked contracts and backward compatibility with contract tests in Pact. The checks ran in both the consumer’s and the provider’s build pipelines, and the contract was published automatically to the Pact server. This applied to HTTP APIs and to messages sent through the messaging system.
  • Ninety percent of our microservices were written in Kotlin on Spring Boot. That made it easier to share what we’d learned about connecting to infrastructure, Azure and monitoring. Once we solved something in one service, we could copy the solution to the others. Microservices do let you try different technologies, but there’s a cost: moving people between teams gets harder and you keep having to work things out from scratch. On our team, hardly anyone wanted to go back to Java after using Kotlin.
  • We stuck strictly to our rule against sharing code between services. The only dependencies between services were network APIs with contracts.
  • We followed the Twelve-Factor App recommendations. Services were configured with environment variables. Logs went to standard output through a console appender, then through the Outcold collector to Splunk.
  • Teams owned the architecture inside their services. They could choose JPA, Hibernate, iBatis or JDBC for data access. We set no limits on service size or complexity and prescribed neither tests nor coverage. The one rule was that public APIs had to be documented and covered by contract tests. We wanted to avoid central teams that dictated architecture, measured quality and so on. The rule was: you design it, write it, test it, deploy it and then operate it. What you do inside is up to you. If a service got out of hand, we assumed the contract would make it straightforward to rewrite and replace.
  • Each service had its own GitLab repository and build pipeline. The pipeline produced a Docker image and scanned for third-party vulnerabilities. Compared with the trouble I remember having with Jenkins at Zonky, GitLab pipelines worked brilliantly. GitLab let us enforce our change management rules centrally. For example, direct commits to master were prohibited and changes had to go through a merge request.
  • Integrating GitLab, Jira and Argo for Kubernetes deployments gave us an adequate audit trail: who did what and why.
  • Most testing happened at the service level. A service was always tested in isolation before its Docker image was built. There was only one environment, Stage, where services came together and a battery of tests ran around the clock. We called them journeys, as described in Testing Microservices. We ran them through the mobile app in an emulator and wrote them in BDD style.
@journey
Feature: Journeys which stop backend production pipeline
Scenario: Top up account | Team A | top_up_account
 Given I type mPIN 1234
 And I’m on ‘Product detail’ screen
 Then I tap on ‘Top up account’ button
Scenario: Create new savings account | Team B | new_savings_account
 Given I type mPIN 1234
 And I’m on ‘Product detail’ screen
 When I tap on ‘Add account’ button
 And I tap on ‘Savings account’ button
 And I tap on ‘Continue’ button
 And I type ‘My savings’ account name
 And I tap on ‘Continue’ button
 And I type mPIN 1234
 Then I’m on ‘Success’ screen
Scenario: Sending a simple future payment to own account | Team A | send_future_payment
 Given I type mPIN 1234
 And I’m on ‘Product list’ screen
 And I tap on ‘my primary’ account
 And I’m on ‘Product detail’ screen
 When I tap on ‘New payment’ button
 And I tap on the ‘My savings’ account
 And I type amount of 42 EUR
 And I tap on ‘Add more details’ button
 And I switch to ‘Future’ payment term
 And I tap on ‘Pay on’ button
 And I select ‘1st of next month’ due date
 And I tap on ‘Continue’ button
 And I tap on ‘Confirm payment’ button
 And I’m on ‘Success’ screen
 And I tap on ‘OK’ button
 Then I’m on ‘Product detail’ screen
 And I tap on ‘Scheduled transfers’ button
 And I tap on first future transaction

If the tests failed, PagerDuty raised an incident immediately. As long as tests on Stage were failing, nobody could deploy to production. We called it the gate. People were understandably frustrated that they could block one another. We couldn’t reliably tell whom to block, and we ran out of time to tune the heuristic.

We released by Kubernetes namespace. Each namespace was effectively a release train for a group of microservices that made up a part of the system, such as risk or payments. Argo handled deployment.

Under the hood we had Azure Container Registry. The promotion pipeline waited 15 minutes, long enough for every journey test to run, checked PagerDuty for an incident indicating a test failure, and tagged the Docker images. Argo then deployed them. We also planned to verify signatures and confirm that Twistlock had completed its vulnerability scan.

Release trains didn’t block one another or require coordination. Teams could deploy independently, and we had 20 to 30 deployments a day. Things still broke occasionally. As far as I recall, none of the more serious problems came from this deployment model.

Load testing

During the final three months we ran regular load tests. Over 24 hours we simulated the operations we expected from the following year’s user base, including daily peaks. This helped us find performance problems and learn how to scale the system and its cloud resources without losing sight of cost. The services and their architecture needed to account for running in the cloud, ideally allocating resources in proportion to load.

We used Kubernetes pod autoscaling so the number of pods could rise and fall with demand. We were just getting started. Scaling pods properly is one thing. Scaling AKS nodes, in Azure’s managed Kubernetes service, is another. Dynamic database scaling ahead of regular month-end or year-end batches was a separate question. We didn’t get much chance to experiment with it and always scaled the database up and down manually.

Chaos engineering experiments

We ran chaos engineering experiments to check stability and availability. The team would make a hypothesis about how the system would behave during a particular failure, then test it. For example, the initial mobile banking dashboard was made up of several independent parts. If one backend component went down, the dashboard was supposed to keep working. We made a hypothesis and tried it. Things didn’t quite behave as expected, so we fixed them. We ran quite a few experiments like that.

Synchronous vs. asynchronous communication

Initially, most communication was synchronous over HTTP. We needed looser coupling between parts of the system, so we moved to asynchronous communication. On Azure we used a message bus for queues and Event Hubs for Kafka-style publish/subscribe. The fun started when we began load testing.

The subsystems were loosely coupled, but they failed under load because parts of the system got overloaded. There were lots of less important messages of type X and relatively few important ones of type Y. We had to split X and Y into separate queues and throttle producers and consumers. We also needed monitoring to see whether it worked. Making this change involves much more than swapping an HTTP controller for a messaging controller. It takes load balancing, monitoring and an understanding of how the messaging middleware works under the hood.

Eventual consistency

We designed the system for availability and speed. For speed we wanted to use Google Firestore, which David Vávra discusses in CZ Podcast episode 214. Here’s an example. At login, we fetched data from the source systems with synchronous API calls and stored a derived version in a local Redis cache with a 24-hour TTL. From there we loaded it into Firestore, to which the reactive mobile app subscribed. Firestore delivered the data quickly to the phone, where it was cached again in Firestore’s local copy. We didn’t have to implement WebSockets or similar plumbing on our side to speed up delivery.

When something happened in the system, such as a card payment, the source system produced an event. In simplified terms, the service responsible for Firestore picked it up and updated the state. This worked beautifully until events got stuck, for instance because of a local overload, or disappeared somewhere. We had to add reconciliation algorithms so that the system would eventually become consistent. Practically every piece of data needed a version or timestamp to let us reconcile the system to its current state. You need to think about that from the start.

Google Firestore

I’ve already touched on this. The technology looks incredibly appealing. In one demo we showed our product team what Firestore made possible: we made a real-time payment on one device and it showed up immediately on another.

Using Firestore means rethinking how you write applications and how you structure them, on the backend as well as the frontend. It’s a different world from a web API. Whether you use REST, GraphQL or RPC, you know how to handle versioning and there are established practices. With Firestore you need to work it out yourself. Imagine, in simplified terms, integrating two systems through a database full of JSON documents.

We used Firestore for reads and GraphQL mutations for writes. In a simple case like renaming an account, the app sent a synchronous GraphQL mutation and used its success or failure to decide whether to show the value the user entered. The new name only appeared in Firestore after the asynchronous messages had arrived and the state had been updated. If that took long enough for someone to close and reopen the app, they would see the old name. We had to come up with patterns for using Firestore in the mobile app, on the server and in the UI. The same applied to performance and data versioning. I think we redesigned our Firestore data structure three times.

Because the customer data in Firestore was sensitive, we chose end-to-end encryption. The server encrypted the data when writing it, and the app decrypted it with the customer’s key. If someone got into Firestore, the data would be useless to them without each customer’s decryption key. We were a little worried about the performance cost, but tests on low-end devices showed that it wasn’t a problem.

Our mobile developers loved Firestore because it made development so easy. You have state, you bind the UI to it, and you’re done. I think it gave the backend team gray hair, especially when the data on screen was stale. You had to check the source of truth, then the cache holding the derived data, then Firestore itself. Since Firestore data was encrypted with a key per customer, we had to write our own admin tool to display the decrypted data.

That might sound like a negative assessment of Firestore, but it’s practically impossible to write a reactive UI without polling, WebSockets or something similar. Firestore did that job extremely well. We ran into the read-scaling problem it had shielded us from when we built online banking. We wanted at least one channel to keep working if Google had an outage, so online banking used GraphQL instead. The server took quite a beating.

Reactive programming

The customer-facing services that populated Firestore or handled GraphQL calls followed the Backend for Frontend pattern. They were written in reactive Kotlin using Reactor. I didn’t work on that code myself, but from what I saw and heard, reconstructing the application’s state without a stack trace was a nightmare. The team got the hang of it eventually. The trouble came when we needed to bring a lot of other developers in. I think only a handful could really write reactive code well. In hindsight I wonder whether it was worth it, given the cost of CPU time versus developer time and the barrier we created for people from other teams who wanted to contribute.

Kotlin

Choosing Kotlin was one of our most important and best decisions. It helped us a lot with hiring because it appealed to Java developers. I can’t say whether it made the code more readable or actually reduced boilerplate.

Observability platform

We built this platform to monitor and operate the entire bank. It supported events, metrics and timely notifications. The events logged by microservices went into Splunk. We built dashboards, reports and alerts on top of it. At peak we logged 600 GB a day and sized the Splunk cluster accordingly.

I was used to heavy Splunk use at GoodData, but Velvon took it to another level. Splunk was the backbone of our monitoring infrastructure and we had to keep tuning it, mainly for performance. Dashboard and report configuration lived in Git, and a GitLab pipeline deployed it to Splunk. We collected the metrics exposed by running services with Prometheus and stored them in VictoriaMetrics. We used Grafana to visualize them.

Conclusion

We could talk about our experiences for a long time. If you’re interested in details I may no longer remember, try contacting the following colleagues. You can also write to me and I’ll point you to someone.

I promised to share what I’m going to do next, but that will have to wait until another time.

Medium version

Related articles

Related CZ Podcast episodes