Managing the Data Volume with MongoDB
Avakash Dekavadiya
MongoDB is a cross-platform and schema less database. It provides high performance, high availability, and easy scalability. It works on concept of collectioan and document.Collection is equivalent of a RDBMS table while a document is a set of key-value pairs, just like tuple/row in RDBMS.It stores data in BSON which is binary JSON.
Sharding is the mechanism of storing data across various machines to meet the needs of growing data. A single machine may not be enough to store data when data grows exponentially. Sharding solves this growing data problem with horizontal scaling. It connects multiple machines to support the data volume and address the challenge of multiple read and write operations simultaneously.
-
Shard Server :-
A shard is a MongoDB physical instance which stores data records. Shard server can be either a single MongoDB instance or a replica set.
-
Config Server :-
Config servers are important MongoDB instances which store the metadata for a sharded cluster. It maps App server to Shard server. So, it is recommended to use three config servers for production deployments.
-
App Server :-
It connects Users with Shard server using metadata of Config server which decide which Shard server to connect. App server also executes read/write operation on sharded clusters.If you want to divide client request, then use multiple App server which is more reliable for robust applications. Multiple users can access simultaneously.
Fig.1 Sharding Server
Replication provides redundancy and increases data availability.There are two kinds of sever in it: primary and secondary. We can write data into primary and only read from secondary and acts during Automatic Failover. When a primary does not communicate with the other members of the set for more than 10 seconds then secondary node becomes primary node automatically.So replication helps our application to be redundant without failure.
Fig. 2 Replication
Sharding and Replication both are combined to harness the benefits of both the features of MongoDB. Sharding addresses data growth and Replication provides for automatic failure recovery.
To configure Sharding and Replication in local, we will demonstrate with one App server, one Config server, two Shard servers and one Replica server corresponding to each primary server. For production environment, it is ideal to use more than one App server to distribute client side requests. More than three Config servers are required in production to avoid single point failure. If the Config server is inaccessible, the cluster is not accessible. More than two shard must be configured or add more shards according to your application. It is recommended to use at least two replica set for each primary node in production for failure recovery.
Fig. 3 Sharding and Replication
-
Create Config server
-
mongod –configsvr –port 27010 –dbpath
-
D:/Mongo_Cluster_Example/configDb
-
mongod –configsvr –port 27010 –dbpath
-
D:/Mongo_Cluster_Example/configDb
-
mongod –configsvr –port 27010 –dbpath
-
D:/Mongo_Cluster_Example/configDb
-
-
Create App server
-
mongod -configdb localhost:27010 –port 27011
-
mongod -configdb localhost:27010 –port 27011
-
mongod -configdb localhost:27010 –port 27011
-
-
Create Shard 1
-
MongoDB -shardsvr –replSet shard1 –port 27012 –dbpath D:/Mongo_Cluster_Example/db1
-
MongoDB -shardsvr –replSet shard1 –port 27012 –dbpath D:/Mongo_Cluster_Example/db1
-
MongoDB -shardsvr –replSet shard1 –port 27012 –dbpath D:/Mongo_Cluster_Example/db1
-
-
Create Shard 2
-
mongod -shardsvr –replSet shard 2 –port 27014 –dbpath D:/Mongo_Cluster_Example/db2
-
mongod -shardsvr –replSet shard 2 –port 27014 –dbpath D:/Mongo_Cluster_Example/db2
-
mongod -shardsvr –replSet shard 2 –port 27014 –dbpath D:/Mongo_Cluster_Example/db2
-
-
Create ReplSet 1
-
mongod –replSet shard1 –port 27013 –dbpath D:/Mongo_Cluster_Example/db1A
-
mongod –replSet shard1 –port 27013 –dbpath D:/Mongo_Cluster_Example/db1A
-
mongod –replSet shard1 –port 27013 –dbpath D:/Mongo_Cluster_Example/db1A
-
-
Create ReplSet 2
-
mongod –replSet shard 2 –port 27015 –dbpath D:/Mongo_Cluster_Example/db2A
-
mongod –replSet shard 2 –port 27015 –dbpath D:/Mongo_Cluster_Example/db2A
-
mongod –replSet shard 2 –port 27015 –dbpath D:/Mongo_Cluster_Example/db2A
-
-
Connect replica set 1 to primary shard
-
mongo –host localhost –port 27012
config = {_id: “shard1”, members: [ { _id: 0, host: “localhost:27012”} , { _id: 1, host: “localhost:27013”} ] }
rs.initiate(config)
rs.status()
-
mongo –host localhost –port 27012
config = {_id: “shard1”, members: [ { _id: 0, host: “localhost:27012”} , { _id: 1, host: “localhost:27013”} ] }
rs.initiate(config)
rs.status()
-
-
Connect replica set 2 to primary shard
-
mongo –host localhost –port 27014
config = {_id: “shard2”, members: [ { _id: 0, host: “localhost:27014”} , { _id: 1, host: “localhost:27015”} ] }
rs.initiate(config)
rs.status()
-
mongo –host localhost –port 27014
config = {_id: “shard2”, members: [ { _id: 0, host: “localhost:27014”} , { _id: 1, host: “localhost:27015”} ] }
rs.initiate(config)
rs.status()
-
mongo –host localhost –port 27014
config = {_id: “shard2”, members: [ { _id: 0, host: “localhost:27014”} , { _id: 1, host: “localhost:27015”} ] }
rs.initiate(config)
rs.status()
-
-
Set App Server metadata
-
mongo –port 27011 –host localhost
sh.addShard( “shard1/localhost:27012,localhost:27013” )
sh.addShard( “shard2/localhost:27014,localhost:27015” )
use mongodb;
db.createCollection(“clustering”)
sh.enableSharding(“mongodb”)
sh.shardCollection(“mongodb.clustering”, {“_id” : 1})
for ( i = 1; i < 999; i++ ) {
db.clustering.insert({ type: “sharding”, score : i});
db.clustering.insert({ type: “replication”, score : i });
}
sh.status()
-
mongo –port 27011 –host localhost
sh.addShard( “shard1/localhost:27012,localhost:27013” )
sh.addShard( “shard2/localhost:27014,localhost:27015” )
use mongodb;
db.createCollection(“clustering”)
sh.enableSharding(“mongodb”)
sh.shardCollection(“mongodb.clustering”, {“_id” : 1})
for ( i = 1; i < 999; i++ ) {
db.clustering.insert({ type: “sharding”, score : i});
db.clustering.insert({ type: “replication”, score : i });
}
sh.status()
-
mongo –port 27011 –host localhost
sh.addShard( “shard1/localhost:27012,localhost:27013” )
sh.addShard( “shard2/localhost:27014,localhost:27015” )
use mongodb;
db.createCollection(“clustering”)
sh.enableSharding(“mongodb”)
sh.shardCollection(“mongodb.clustering”, {“_id” : 1})
for ( i = 1; i < 999; i++ ) {
db.clustering.insert({ type: “sharding”, score : i});
db.clustering.insert({ type: “replication”, score : i });
}
sh.status()
-
-
Check in shard 1 PRIMARY
-
mongo –host localhost –port 27012
use mongodb
db.clustering.find().sort({_id : 1}).limit(1).pretty()
-
mongo –host localhost –port 27012
use mongodb
db.clustering.find().sort({_id : 1}).limit(1).pretty()
-
-
Check in shard 1 SECONDARY
-
mongo –host localhost –port 27013
rs.slaveOk()
use mongodb
db.clustering.find().sort({_id : 1}).limit(1).pretty()
-
mongo –host localhost –port 27013
rs.slaveOk()
use mongodb
db.clustering.find().sort({_id : 1}).limit(1).pretty()
-
-
Check in shard 2 PRIMARY
-
mongo –host localhost –port 27014
use mongodb
db.clustering.find().sort({_id : 1}).limit(1).pretty()
-
mongo –host localhost –port 27014
use mongodb
db.clustering.find().sort({_id : 1}).limit(1).pretty()
-
-
Check in shard 2 SECONDARY
-
mongo –host localhost –port 27015
rs.slaveOk()
use mongodb
db.clustering.find().sort({_id : 1}).limit(1).pretty()
-
mongo –host localhost –port 27015
rs.slaveOk()
use mongodb
db.clustering.find().sort({_id : 1}).limit(1).pretty()
-
Here, Step 1 to 6 are for connecting all necessary nodes. Steps 7 , 8 and 9 for connecting them together. Step 10 to 13 are for testing your configuration. You will find same data in shard1 and it’s replica set, similarly for shard 2 and it’s replica set. Also data which is inserted through App server are distributed across both shard servers.
Concept of connected devices and better internet services are creating volumes of data every nano second. It’s imperative for organizations to ensure quick access to data while maintaining server uptime across 24 hours. MongoDB comes to rescue with its sharding and replication features that addresses both the issues: quick access and server uptime.
Our team of experts guide you to set up data servers to manage your ever growing data needs. Call us to know more about our MongoDB expertise.
