MongoDB Sharding - Concepts, Examples and Tutorials

Sharding in mongodb

Sharding helps you to improve the performance,scalability and availablity of database applications.

When you're dealing with massive amounts of data or apps that need to handle tons of requests, a single server can start to feel the pressure.

For instance, if the system gets flooded with too many queries at once, the server's CPU can quickly get overwhelmed.

And if the active data set is bigger than the server's available memory, the system has to rely more on the hard drive, which can really slow things down.

To keep up with growing demands, systems can scale in two main ways: by upgrading the existing machine (vertical scaling) or by adding more machines to share the load (horizontal scaling).

Vertical Scaling

• Vertical scaling means boosting the power of a single server—like upgrading to a faster CPU, adding extra RAM, or expanding storage.

• While this can give your system a performance boost, there’s a catch: every machine has its limits.

• Even cloud providers can only offer so much hardware, and once you hit those limits, you can't scale any further. That’s why vertical scaling eventually reaches a ceiling.

Horizontal Scaling

• Horizontal scaling is all about spreading the workload across multiple servers.

• Instead of relying on one super-powerful machine, you bring in more servers to share the job.

• Each server handles a piece of the overall data or traffic, which can actually be more efficient—and often cheaper—than investing in a single high-end machine.

• The beauty of this approach is flexibility: need more power? Just add more servers. The downside?

• It adds complexity to your system setup and maintenance, so managing everything becomes a bit trickier.

Note: MongoDB supports horizontal scaling through sharding.

Advantages of Sharding

• Improved Read/Write Performance

  In a sharded MongoDB cluster, the workload gets split across multiple shards, so each server only handles a portion of the total read and write traffic.

  As your app grows, you can simply add more shards to handle the increased load—scaling out instead of up.

  If your queries include the shard key (or start with it, in the case of compound keys), MongoDB can send that request directly to the right shard instead of asking every shard. This targeted approach makes those queries much faster and more efficient.

• More Storage, Seamlessly

  Sharding spreads your data across several machines, so each shard stores just a slice of the overall dataset.

  As your data grows, all you have to do is add more shards, and your cluster’s total storage capacity grows with it. No need for risky migrations or giant single-server upgrades.

• Built-In High Availability

  Each shard and config server in a MongoDB cluster is set up as a replica set.

  This means if a few nodes go offline, the system still stays up and running.

  Even if a whole shard becomes unreachable, the cluster can continue processing reads and writes for the remaining shards, so your app doesn’t grind to a halt.

How to set up a MongoDB sharding

If you're planning to deploy a MongoDB sharded cluster across multiple machines, here's a straightforward walkthrough to help you set it up using three Ubuntu 20.04 LTS servers, each pre-installed with MongoDB version 4.4.1.

For clarity, let's label each server based on its role in the cluster:

mongodb01 (10.10.10.56) – This machine will act as the Config Server, managing metadata and cluster configuration.

mongodb02 (10.10.10.57) – This one will function as the Query Router, or mongos, responsible for routing queries to the appropriate shards.

mongodb03 (10.10.10.58) – This server will serve as a Shard, storing a subset of the data in the cluster.

🛠 Step 1: Set Up a Dedicated Directory for the Config Server

• you’ll need to set up a specific folder on your Config Server to store its data. Think of this as giving MongoDB a clean space to work with.

• If you're on mongodb01 (10.10.10.56), open your terminal and run the following command to create a directory:

sudo mkdir -p /data/configdb

• The -p flag ensures that the entire path is created if it doesn’t already exist. This folder will hold all the configuration data needed to keep your sharded cluster in sync.

⚙️ Step 2: Launch MongoDB as a Configuration Server

• it’s time to start MongoDB in configuration mode on our Config Server.

• Let’s assume your config server is running on mongodb01 (10.10.10.56). You can spin up MongoDB as a Config Server using this command:

mongod --configsvr --replSet configReplSet --dbpath /data/configdb --port 27019 --bind_ip 0.0.0.0 --fork --logpath /var/log/mongodb/config.log

• Here’s a quick breakdown of what each option means:

  configsvr: Tells MongoDB that this instance is a configuration server.

  replSet configReplSet: We're putting the config server into a replica set named configReplSet. Even a single config server must be in a replica set in sharded clusters.

  dbpath: Points MongoDB to the directory we created earlier.

  port 27019: Standard port for config servers, but you can choose a different one if needed.

  bind_ip 0.0.0.0: Makes the service accessible from any IP (useful during setup; you can tighten security later).

  fork: Runs the service in the background.

  logpath: Specifies where MongoDB should store its log file.

🌐 Step 3: Fire Up the Mongos (Query Router) Instance

• With your Config Server up and running, the next step is to start the mongos process—this is the query router that acts as the gateway between your application and the sharded cluster.

mongos --configdb configReplSet/10.10.10.56:27019 --bind_ip 0.0.0.0 --port 27017 --fork --logpath /var/log/mongodb/mongos.log

• configdb configReplSet/10.10.10.56:27019: This tells mongos where to find the config server and what the replica set is called (in this case, configReplSet).

• bind_ip 0.0.0.0: Allows connections from any IP—helpful during development or setup.

• port 27017: This is the default MongoDB port, which makes it easier for apps to connect without custom configuration.

• fork: Runs the process in the background.

• logpath: Defines where to save the mongos logs.

🔗 Step 4: Connect to the Mongos Instance

• Now that your mongos router is up and running, it’s time to connect to it using the MongoDB Shell. This will let you interact with your sharded cluster and begin adding shards or databases.

• Hop onto the terminal (this can be from the same machine or a different one that can reach your query router) and run:

mongo --host 10.10.10.57 --port 27017

host 10.10.10.57: Points to your mongos server (which is on mongodb02).

port 27017: Connects to the port your mongos is listening on (default is 27017).

🧩 Step 5: Add Shard Servers to the Cluster

• Now that you’re connected to the mongos router, it’s time to plug in your shard servers and officially grow your cluster!

sh.addShard("10.10.10.58:27018")

• sh.addShard(...): This command registers a new shard with the cluster.

• 10.10.10.58:27018: The IP and port of your shard server. Make sure the shard is already running a mongod instance and listening on that port.

sh.addShard("shardReplSet/10.10.10.58:27018")

🛡 Step 6: Set Up Replica Sets for Your Shard Servers

• To make your sharded cluster more resilient and production-ready, it’s a good idea to turn each shard into a replica set. This ensures high availability and fault tolerance in case one of your nodes goes down.

• Assume you're on your shard server (e.g., mongodb03 – 10.10.10.58) and you've already started the mongod process with replication enabled. Now, open the Mongo shell on that server and run the following command to initiate the replica set:

rs.initiate({
  _id: "shardReplSet",
  members: [
    { _id: 0, host: "10.10.10.58:27018" }
  ]
})

_id: "shardReplSet": This names your replica set. Make sure it matches the name you’ll use when adding the shard to the cluster via mongos.

members: This is where you list all the nodes in your replica set. For now, we’re just starting with one, but you can add more later.

🧱 Step 7: Link Shards to the Cluster via Mongos

At this point, your shard servers might be up and running—maybe even configured as replica sets—but they're still floating outside the actual sharded cluster. To bring them into the cluster and make them official members, you'll need to use the mongos router to add them in.

Head back to the Mongo shell connected through your mongos instance (on mongodb02), and use the following command to register your shard:

sh.addShard("shardReplSet/10.10.10.58:27018")

"shardReplSet" is the name of the replica set you configured in Step 6.

"10.10.10.58:27018" is the address and port of the primary node in that replica set.

📂 Step 8: Enable Sharding for a Specific Database

Now that your sharded cluster is fully wired up and all the pieces are in place, it’s time to turn on sharding for the actual database you want to distribute across your shards.

To do this, hop into the Mongo shell connected to your mongos instance (on mongodb02) and run the following command:

sh.enableSharding("yourDatabaseName")

Replace "yourDatabaseName" with the name of the database you want to shard. For example, if you’re working with a database called inventory, the command would look like this:

sh.enableSharding("inventory")

What is the composition of ObjectID ?

ObjectID is composed of a timestamp, client machine ID,client process ID, and a 3-byte incremented counter.

What are the indexes in MongoDB ?

Indexes in MongoDB improve query performance

It enables the database to quickly locate documents based on the indexed fields.

Code Blocks

db.collection.createIndex({ "age": 1 });  // age_1

db.collection.createdIndex({ "age" : 1, "gender" :1 }) // age_1_gender_1

Explanation:-

First one code will create age index and age: 1 means it will index with ascending order, if we write -1 then it will do it descending order

Second code will create mulitple index with age and gender