In today’s data-driven world, where the volume and complexity of data continue to expand at an unprecedented pace, the need for robust and scalable database solutions has become paramount. It is estimated that 180 zettabytes of data will be created by 2025. Those are big numbers to wrap your head around. As data and user demand skyrocket, relying on a single database location becomes impractical. It slows down your system and overwhelms developers. You can adopt various solutions to optimize your database, such as database sharding. In this comprehensive guide, we delve into the depths of MongoDB sharding, demystifying its benefits, components, best practices, common mistakes, and how you can get started.
What Is Database Sharding?
Database sharding is a database management technique that involves partitioning a growing database horizontally into smaller, more manageable units known as shards. As your database expands, it becomes practical to divide it into multiple smaller parts and store each part separately on different machines. These smaller parts, or shards, are independent subsets of the overall database. This process of dividing and distributing data is what constitutes database sharding.
Build vs Buy for a Sharding Solution
When implementing a sharded database, there are two primary approaches: developing a custom sharding solution or paying for an existing one. This raises the question of whether building a sharded solution or paying is more suitable.
To make this choice, you need to consider the cost of 3rd party integration, keeping in mind the following factors:
- Developer skills and learnability: The learning curve associated with the product and how well it aligns with the skills of your developers.
- The data model and API offered by the system: Every data system has its own way of representing its data. The convenience and ease with which you can integrate your applications with the product is a key factor to consider.
- Customer support and online documentation: In cases where you may encounter challenges or require assistance during integration, the quality and availability of customer support and comprehensive online documentation become crucial.
- Availability of cloud deployment: As more companies transition to the cloud, it is important to determine whether the third-party product can be deployed in a cloud environment.
Based on these factors, you can now decide to either build a sharding solution or pay for a solution that does the heavy lifting for you. Today, most of the databases in the market support database sharding. For instance, relational databases like MariaDB (a part of the high-performance server stack at Kinsta) and NoSQL databases like MongoDB.
What Is Sharding in MongoDB?
The primary purpose of using a NoSQL database is its ability to deal with the computing and storage demands of querying and storing humongous volumes of data. Generally, a MongoDB database contains a large number of collections. Every collection consists of various documents that contain data in the form of key-value pairs. You can break up this large collection into multiple smaller collections using MongoDB sharding. This allows MongoDB to perform queries without putting much strain on the server.
For example, Telefónica Tech manages over 30 million IoT devices worldwide. To keep up with the ever-increasing device usage, they needed a platform that could scale elastically and manage a fast-growing data environment. MongoDB’s sharding technology was the right choice for them since it was the best fit for their cost and capacity needs. With MongoDB sharding, Telefónica Tech runs well over 115,000 queries per second. That’s 30,000 database inserts per second, with less than one millisecond of latency!
Benefits of MongoDB Sharding
Here are a few benefits of MongoDB sharding for large-scale data that you can enjoy:
- Storage Capacity: We’ve already seen that sharding spreads the data across the cluster shards. This distribution lets each shard contain a fragment of the total cluster data. Extra shards would increase the cluster’s storage capacity as and when your data set grows in size.
- Reads/Writes: MongoDB distributes read-and-write workload across shards in a sharded cluster, allowing each shard to process a subset of cluster operations. Both workloads can be scaled horizontally across the cluster by adding more shards.
- High Availability: The deployment of shards and config servers as replica sets offer increased availability. Now, even if one or more shard replica sets become completely unavailable, the sharded cluster can perform partial reads and writes.
- Protection From an Outage: Many users get affected if a machine bites the dust due to an unplanned outage. In an unsharded system, since the whole database would have gone out, the impact is massive. The blast radius of bad user experience/impact can be contained through MongoDB sharding.
- Geo-Distribution and Performance: Replicated shards can be placed in different regions. This means that customers can be provided with low-latency access to their data i.e., redirect consumer requests to the shard nearer to them. Based on the data governance policy of a region, specific shards can be configured to be placed in a specific region.
Components of MongoDB Sharded Clusters
Having explained the concept of a MongoDB sharded cluster, let’s delve into the components that comprise such clusters.
1. Shard
Every shard has a subset of the sharded data. As of MongoDB 3.6, shards must be deployed as a replica set to provide high availability and redundancy. Every database in the sharded cluster has a primary shard that’ll hold all the unsharded collections for that database. The primary shard isn’t related to the primary in a replica set. To change the primary shard for a database, you can use the movePrimary command. The primary shard migration process might take a significant time to complete. During that time, you shouldn’t attempt to access the collections associated with the database till the migration process is completed. This process might impact overall cluster operations based on the amount of data being migrated. You can use the sh.status() method in mongosh to look at the cluster’s overview. This method will return the primary shard for the database along with the chunk distribution across the shards.
2. Config Servers
Deploying config servers for sharded clusters as replica sets would improve the consistency across the config server. This is because MongoDB can leverage the standard replica set read and write protocols for the config data. To deploy config servers as a replica set, you’ll have to run the WiredTiger storage engine. WiredTiger uses document-level concurrency control for its write operations. Therefore, multiple clients can modify different documents of a collection at the same time. Config servers store the metadata for a sharded cluster in the config database. To access the config database, you can use the following command in the mongo shell: use config. Here are a few restrictions to keep in mind here:
- A replica set configuration used for config servers should have zero arbiters. An arbiter participates in an election for the primary, but it doesn’t have a copy of the dataset and can’t become the primary.
- This replica set cannot have any delayed members. Delayed members have copies of the replica set’s dataset. But a delayed member’s data set contains an earlier or delayed state of the data set.
- You need to build indexes for the config servers. Simply put, no member should have members[n].buildIndexes setting set to false.
- If the config server replica set loses its primary member and cannot elect one, the cluster’s metadata becomes read-only. You’ll still be able to read and write from the shards, but no chunk splits, or migration will occur until the replica set can elect a primary.
3. Query Routers
MongoDB mongos instances can serve as query routers, allowing client applications and the sharded clusters to connect easily. Starting in MongoDB 4.4, mongos can support hedged reads to decrease latencies. With hedged reads, the mongos instances will dispatch read operations to two replica set members for every shard that’s queried. It’ll then return results from the first respondent per shard.
Here’s how the three components interact within a sharded cluster:
A mongos instance will direct a query to a cluster by:
- Checking the list of shards that need to receive the query.
- Establish a cursor on all targeted shards. The mongos will then merge the data from each targeted shard and return the result document. Some query modifiers, like sorting, are executed…