Sharding — Split up your data

Horizontal data partitioning, or sharding, is a way to scale with large volumes of data. It has been used successfully in the architecture of Google applications, Flickr and Friendster. The idea is simple. Use a key to distribute data among several physical stores or databases. Think of a dating app. You could distribute the data in different ways.

• User data for A–G goes in database A, user data for H–R in database B, and so on.
• User information goes in database A, conversation history in database B, and public discussion forums in database C.
• All data for users in Czechia goes in database A, and all data for users in Slovakia goes in database B.

Sharding gives us several advantages.

• The application works with a limited, relatively small amount of data at a time, so it can read and write faster.
• If one database machine, or shard, goes down, the whole service doesn’t go down. Only the data on that machine is affected.
• For high availability, you only need to replicate the shard in question.
• You can process data in parallel. For example, you can run the same query on several shards at once.

Of course, sharding also has disadvantages.

• Data isn’t normalized across shards, so you can’t easily query several of them together. You have to do that in code by running the query on every shard and combining the results.
• You need a strategy for finding which shard contains the data.
• Sharding can’t be transparent. The data access logic has to know the data is split.

Apart from Hibernate Shards, which still seems a little rough to me, you’ll have to cobble together the whole thing yourself. Imagine implementing it in JDBC. It certainly won’t be a two-day job. On the other hand, you’ll have full control without depending on database features that cost a fortune. Once you start partitioning data horizontally, you have to change how you work with it.

I’d be curious to know whether high-traffic Czech portals such as Seznam or Aukro use sharding.

I drew heavily on An Unorthodox Approach to Database Design: The Coming of the Shard for this post and highly recommend reading it.