Scalability on paper, bcrypt and debugging Zonky's performance

Lately I have often been answering technical questions from our investors about Zonky getting overloaded when high-interest loans appear, which regularly sparks a scramble to invest. I would like to put the problems and the fixes we made in one place.

Our solutions inevitably depend on the application architecture and the resources available to us. Zonky runs on a classic Java stack: Spring, JPA and Postgres as its database. Like any startup, we first validated the business model and its growth instead of tuning the architecture. To my mind that is entirely valid and the only right approach. What good is a super-scalable architecture if you have no paying users? A rhetorical question, of course.

Our architecture thus reflects, or did reflect, the fact that we were a startup going to market with an MVP. It was a classic monolith. Vertical scaling, yes. Horizontal scaling, on paper. Why only on paper when the world is full of technologies with seemingly unlimited scaling potential, from Docker, Kubernetes and Redis to the cloud and virtualization?

With a classic Java stack you have the theoretical ability to scale horizontally. In our case, that means deploying the application to multiple machines. Again, this is theory. Anyone who has operated a distributed system knows that things get considerably more complicated once shared state and the network come into play. See the CAP theorem and its implications.

A brief detour to Zonky. For us, shared state is represented by an in-process cache that takes load off the database. The cache holds all the critical data needed to make an investment, including the amount currently invested and authentication tokens.

Back to CAP and its implications. Your code and technologies may look as though they work in a distributed environment, but in practice you may not know what problems are coming your way. Network partitions, slower communication, dropped packets and clocks drifting apart will all reach you sooner or later.

What separates theoretical horizontal scalability from practical horizontal scalability is whether you have accounted for these failure cases. You cannot just say, “My stack supports it, let's go.” You have to look at the shared state and deal with it. No technology will magically solve every shared-state problem for you.

Our traffic is steady, then jumps by an order of magnitude when a high-interest loan appears because timing and speed are critical. One thing in our favor is that we know a little in advance that such an event is coming. Again, that is useful in theory. If you cannot scale out, the information does you no good.

Most of our early performance issues were easy to fix at the database. We added a missing index, optimized a JPA query or applied another familiar ORM optimization. Some time ago, though, we ran into a problem that took considerably more ingenuity.

The application would consume all available CPU, hang briefly, then resume. The telemetry pointed to bcrypt, which we use to hash passwords. Because of how long authentication tokens were valid, hundreds of users tried to log in at once. As a secure password hashing function, bcrypt is slow by design.

What made no sense was the application freezing and then resuming on its own. The telemetry showed all database connections in use. A closer look found the culprit in the OpenSessionInView antipattern combined with a cache side loader.

A database connection stayed active for the whole request. Once the CPU was saturated, threads that did not actually need their database connections because they were waiting for CPU time consumed all the connection capacity. Eventually they either got their turn or timed out. At that point the whole system came back to life. How?

Our workload is read-heavy, so we serve practically all traffic from the cache. A side loader populates the cache (Hazelcast) from the database and runs in a dedicated thread pool. The application threads waiting on the cache and the side-loader threads loading it from the database exhausted the available connections. The side loaders were waiting for connections needlessly held by application threads because of OpenSessionInView, while those threads waited for CPU time fully taken up by bcrypt. When a timeout occurred or a connection became available to a side-loader thread, the system slowly moved forward again.

The order in which we applied our patches:

1. Scale vertically by adding CPU capacity.
2. Defer allocating a database connection until the latest possible moment, typically the start of a transaction.
3. Remove cache side loading for the most common use cases.
4. Give side loaders a dedicated database connection pool so neither side can exhaust the other's connections.
5. Put a semaphore in front of bcrypt to cap concurrency at a level that does not cripple the system (CPU count minus one).

Given the urgency, we could neither remove OpenSessionInView entirely nor rewrite the side loader.

Next we plan to offload bcrypt to spare capacity in our data center. We have a microservice that we will deploy on N machines. Its only job will be hashing, with local computation always available as a fallback.

This raises the question of scaling out to the cloud. A serverless service such as AWS Lambda would be a good fit. We already have a hybrid deployment, with server-side rendering for our SPA running on AWS Elastic Beanstalk. Lambda would have considerably higher latency, and I worry it might not scale elastically to a sudden surge over a few dozen seconds. We might have to warm up the Lambda functions with synthetic traffic to bring enough instances online for the expected load. The infrastructure required would be much more complex than the microservice itself.