A Technical Look Back at 2017
At the start of 2018 we met at K10 coworking to look back at 2017 and take stock of what we’d achieved technically at Zonky.
Rewind the tape
A retrospective naturally starts at the beginning of 2017. We’d just been through the first phase of growing the team. In autumn 2016, all of Zonky still ran on the work of five, yes, five developers. By the start of 2017 there were sixteen of us.
That kind of growth brought plenty of problems. All the technical knowledge was concentrated in Brno, but the team was growing in Prague. It also became clear very quickly that the more developers you have, the more overhead you have.
Code review became a bottleneck because only two developers had the knowledge it required. Unplanned work kept getting underfoot as well, whether it was bug fixes or last-minute changes that needed doing yesterday. We had a ten-day release cycle, and releases were manual to the point of copying commands from Confluence. We had only basic operating system telemetry for CPU, memory, disk and network, and one or two very basic application health checks. Our deployment spanned one virtual host that ran the database, application server and HTTP server together.
The first thing we did was reorganize. The goals were to:
- Spread key knowledge across teams, even though that meant distributed teams. Three developers from Brno joined teams Alfa, Bravo and Charlie, named after Navy SEAL teams because, of course, we have the best developers.
- Dedicate one third of our development capacity to unplanned work. Unplanned work also got its own backlog, and requests came through Service Desk, which is friendlier for ordinary users, in place of JIRA. If you’re interested, you can read about the sweeper team.
- Expand the group of gatekeepers with permission to merge pull requests to include developers who could take some of the simpler requests off their hands.
A ten-day cycle put us in a difficult position whenever a release was in jeopardy. Delaying it another ten days would make testing very risky because of how many changes would pile up. You know how it goes: more developers, more code, or more changes if you prefer. Even though we weren’t technically ready and still released by hand, we moved to weekly releases. If it hurts do it more often. We improved the release process on several fronts.
- We replaced manually copying commands from Confluence with a procedure rewritten entirely in Chef and later in Ansible.
- We also invested in releases without downtime, which let us deploy certain kinds of changes without taking the system down. That deserves an article of its own.
- Smoke tests replaced at least part of the post-release verification we used to do manually.
- For some new product features, such as the secondary market, we separated deployment from rollout to users with feature toggles through LaunchDarkly
Better monitoring and oversight gave us visibility into how the system was running. Application logs go into Splunk, where we can search them while troubleshooting. We’ve configured basic alerts, typically for business operations that should have happened and didn’t, or whose failures appear in the log, such as processing bank transactions or reporting on contracts. We collect application metrics with Coda Hale’s Metricsand store them alongside operating system metrics as time series in InfluxDB and visualize them with Grafana. We check all our health checks, whose number has grown considerably, with Icinga 2. For real-time JVM monitoring and troubleshooting performance problems in production, we use New Relic.
Every developer started taking an on-call shift once every few weeks, following the principle You build it You run it. An overview of system health is available on the operations dashboard.
We moved some tests and development use cases into Docker. Anyone can start a fresh instance of Zonky in Docker from any Git branch at any time. We also moved all test environments to AWS and added a few so that we could test against an environment with a topology as close to production as possible and restore it whenever we needed to.
We often used dynamic scaling in AWS. For example, we increased the number of Jenkins or Docker instances tenfold depending on what developers needed. Capacity grew and shrank as needed, all on Spot Fleet instances with much better prices, at a discount of 80–90%.
One of our biggest problems was Zonky going down when a particularly attractive loan appeared on the marketplace. I described it in detail in this article. In the end we used dynamic scaling again, this time on spare capacity in the data center. As we moved toward microservices, we built an API gateway on Netflix Zuul. It became the entry point for every HTTP request into the system. You can find out more about the API gateway in this video.
All our microservices, fewer than ten for now, run in high-availability mode. To deploy without downtime, we had to review all the application logic to account for request retries. Deployment without downtime was one of our biggest infrastructure efforts. It sounds like a development topic, but it was mainly about governance and changing the flow of the development process.
All services run on Spring Boot and Spring Cloud, which lets us use infrastructure such as the Spring Cloud Config Server or the community project Spring Boot Admin.
One big area, and a source of problems of course, was the rewrite from Angular.js to Ember.js, which deserves an article of its own. We did it because we needed server-side rendering for SEO. We set up deployment for the frontend in AWS on Elastic Beanstalk and optimized it with the CloudFront CDN.
In the meantime, with help from our hosting provider S1, we consolidated the backend deployment into one data center. That included physically moving the bare-metal machine with the production database. On top of that, the mobile application and its development came out of the blue. For the backend, it meant getting to grips with several technologies, including Spring REST Docs (REST tests as API documentation), Spring Cloud Contract (contract tests), and other Spring Cloud Netflix libraries so we could expose an API tailored to the mobile app.
I’m sure I’ve forgotten a lot of things, but I still think that’s a respectable list of what we managed to do in 2017.
For 2018 I chose to focus on improving and automating our tests to support releases without downtime. I wanted to cut the number of releases with downtime to 21 and replace them with ones that don’t interrupt service. We also have more team growth ahead, with all the joys and troubles that brings. On the infrastructure side we’ll take on Kubernetes, and we’ll move the back-office interface to Angular 4.
By the way, we’re still hiring in Prague and Brno. You can learn more about our culture from the engineering blog or from a few older interviews.
- https://www.zonky.cz/zonkytimes/praci-si-musim-zakazovat-rika-martin-pavlik-v-zonky-propojuje-byznys-s-it/
- https://www.zonky.cz/zonkytimes/je-to-daleko-narocnejsi-ale-to-je-ta-spravna-vyzva-rika-chirurg-ze-zonky/
- https://www.zonky.cz/zonkytimes/na-baby-sitting-nejsme-vybaven-ani-mentlne-ani-rychlost-rk-f-vvojru-v-zonky/
- https://www.zonky.cz/zonkytimes/bavi-me-ze-delame-veci-pro-skutecne-lidi-rika-vyvojar-zonky-lukas/
Related articles
Related CZ Podcast episodes
- #261 · Leadership
On leading people and teams. · Czech audio