mongoDB 3.2 replica set cluster with arbiter using docker

How to deploy a mongoDB 3.2 replica set cluster using docker


We will deploy:

  1. MongoDB 3.2.4 with WiredTiger storage engine
  2. Replica set with 2 nodes and 1 arbiter
  3. Authentication enabled
  4. Persistent data to each node local file system

To follow this tutorial you must have docker installed on your servers or VMs. You can find instructions to do so here.
I'll also assume you can run docker without sudo and that you are using Debian or one of its derivatives.

Official mongoDB documentation about replica set can be found here.

Before proceeding, it might be well worth to read MongoDB MMAPv1 vs WiredTiger storage engines, since WiredTiger differs quite significantly from MMAPv1 and might not be the right choice for everyone, depending on use case and hardware.

Step One:

Get the IPs of all three servers running the following command on each machine:

ifconfig eth0:1 | grep "inet addr" | cut -d: -f2 | awk '{print $1}'  

(If you are using a different network interface other than eth0:1, make sure to modify the above command accordingly)

Then export them on every machine:

yourUsername@yourServerName1:~$ export srv1=192.168.207.23  
yourUsername@yourServerName2:~$ export srv2=192.168.204.21  
yourUsername@yourServerName3:~$ export arb=192.168.207.31  

(Make sure to change the IP addresses, to match your servers ones, before exporting.)

In a production environment also make sure each of the servers is accessible by way of resolvable DNS or hostnames. Either set up '/etc/hosts' to reflect this configuration or configure your DNS names.

Step Two:

For this blog post I'll use /home/docker/mongo directory. Create the directory on all servers. On yourServerName1, also cd into it:

yourUsername@yourServerName2:/mkdir -p ~/docker/mongo  
yourUsername@yourServerName3:/mkdir -p ~/docker/mongo  
yourUsername@yourServerName1:/mkdir -p ~/docker/mongo && cd $_  

Create a key file to be used on all of your servers. Do this on yourServerName1:

yourUsername@yourServerName1:~$ openssl rand -base64 741 > mongodb-keyfile  
yourUsername@yourServerName1:~$ chmod 600 mongodb-keyfile  
yourUsername@yourServerName1:~$ chown 999 mongodb-keyfile  

The file owner has to be changed to uid 999 because the user in the mongoDB docker container is the one that needs to access the key file.

Copy mongodb-keyfile on the remaining servers:

scp mongodb-keyfile yourUsername@yourServerName2.yourDomain.com:~/docker/mongo  
scp mongodb-keyfile yourUsername@yourServerName3.yourDomain.com:~/docker/mongo  
Step Three

On yourServerName1, start mongoDB docker container with no authentication:

yourUsername@yourServerName1:~$ docker run --name mongo3.2.4 \  
-v /home/yourUsername/docker/mongo/data:/data/db \
-v /home/yourUsername/docker/mongo:/opt/keyfile \
--hostname="yourServerName1.yourDomain.com" \
-p 27017:27017 \
-d mongo:3.2.4

To connect to the mongoDB docker container we just created, start an interactive shell:

yourUsername@yourServerName1:~$ docker exec -it mongo3.2.4 /bin/bash  

You are now inside the mongo Docker container. Start a mongo shell:

yourUsername@yourServerName1:~$ mongo

MongoDB shell version: 3.2.4  
connecting to: test  
Welcome to the MongoDB shell.  
For interactive help, type "help".  
For more comprehensive documentation, see  
http://docs.mongodb.org/  
Questions? Try the support group  
http://groups.google.com/group/mongodb-user  
>

Switch to the admin database:

> use admin
switched to db admin  

Create a new site admin user:

> db.createUser( {
     user: "siteUserAdmin",
     pwd: "yourSuperS3cretPassword",
     roles: [ { role: "userAdminAnyDatabase", db: "admin" } ]
   });

A successful message will show up:

Successfully added user: {  
"user" : "siteUserAdmin",
"roles" : [
         {
              "role" : "userAdminAnyDatabase",
              "db" : "admin"
         }
      ]
}

Create a new root user:

> db.createUser( {
     user: "siteRootAdmin",
     pwd: "yourSuperS3cretPassword",
     roles: [ { role: "root", db: "admin" } ]
   });

Again, a successful message will show up:

Successfully added user: {  
            "user" : "siteRootAdmin",
                  "roles" : [
            {
            "role" : "root",
                  "db" : "admin"
            }
        ]
}

Those are the users we will be using to admin our replica set.
Exit from both mongoDB interactive shell and docker container:

> exit
bye  
yourUsername@yourServerName1:~$ exit  
Step Four

Stop the first mongoDB docker container instance:

yourUsername@yourServerName1:~$ docker stop mongo3.2.4  
Step Five

Delete the mongoDB docker container we created earlier to create admin and root user to manage our cluster. Fear not, deleting the docker container won't delete our data as well, since we mounted /data/db on a local path. Remember the line -v /home/core/mongo-files/data:/data/db? That's exactly what it does.

yourUsername@yourServerName1:~$ docker rm mongo3.2.4  

Start a new mongoDB docker container on yourServername1, but this time let's use the key file we created earlier:

yourUsername@yourServerName1:~$ docker run \  
--name mongo3.2.4 \
-v /home/yourUsername/docker/mongo/data:/data/db \
-v /home/yourUsername/docker/mongo:/opt/keyfile \
--hostname="yourServerName1.yourDomain.com" \
--add-host yourServerName1.yourDomain.com:${srv1} \
--add-host yourServerName2.yourDomain.com:${srv2} \
--add-host yourServerName3.yourDomain.com:${arb} \
-p 27017:27017 -d mongo:3.2.4 \
--keyFile /opt/keyfile/mongodb-keyfile \
--replSet "rs0"

The --keyFile parameter is the path to the key file inside the mongoDB docker container. With this line: -v /home/core/mongo-files:/opt/keyfile we mapped our local path /home/yourUsername/docker/mongo/ to /opt/keyfile/ inside the mongoDB docker container.

The --add-host is used to edit /etc/hosts inside the mongoDB docker container, so we can use hostnames instead of IPs. In a production environment these entries can be resolved via DNS, so those lines could be skipped.

Step Six

Again on yourServername1, connect to the mongoDB docker container, starting an interactive shell and the start a mongo shell as well and then authenticate as root user we created earlier:

yourUsername@yourServerName1:~$ docker exec -it mongo3.2.4 /bin/bash  
yourUsername@yourServerName1:~$ mongo  
MongoDB shell version: 3.2.4  
> 
Switch to the admin database:  
> use admin
switched to db admin  
> db.auth("siteRootAdmin", "yourSuperS3cretPassword");
1  

Eventually, we can initiate the replica set:

> rs.initiate()
{
         "info2" : "no configuration explicitly specified -- making one",
         "me" : "yourServerName1.yourDomain.com:27017",
         "info" : "Config now saved locally.  Should come online in about a minute.",
         "ok" : 1
}
>
Step Seven

View the initial replica set configuration:

rs0:PRIMARY> rs.conf()  
{
        "_id" : "rs0",
        "members" : [
              {
                  "_id" : 0,
                  "host" : "yourServerName1.yourDomain.com:27017"
              }
        ]
}

(shortened version)

Step Eight

Create and start mongoDB docker container on yourServerName2:

yourUsername@yourServerName2:~$ docker run \  
--name mongo3.2.4 \
-v /home/yourUsername/docker/mongo/data:/data/db \
-v /home/yourUsername/docker/mongo:/opt/keyfile \
--hostname="yourServerName2.yourDomain.com" \
--add-host yourServerName1.yourDomain.com:${srv1} \
--add-host yourServerName2.yourDomain.com:${srv2} \
--add-host yourServerName3.yourDomain.com:${arb} \
-p 27017:27017 -d mongo:3.2.4 \
--keyFile /opt/keyfile/mongodb-keyfile \
--replSet "rs0"

The third one, yourServerName3, is going to be the arbiter, so we are going to use a slightly modified configuration:

yourUsername@yourServerName2:~$ docker run \  
--name mongo3.2.4 \
-v /home/yourUsername/docker/mongo/data:/data/db \
-v /home/yourUsername/docker/mongo:/opt/keyfile \
--hostname="yourServerName3.yourDomain.com" \
--add-host yourServerName1.yourDomain.com:${srv1} \
--add-host yourServerName2.yourDomain.com:${srv2} \
--add-host yourServerName3.yourDomain.com:${arb} \
-p 30000:27017 -d mongo:3.2.4 \
--nojournal \
--quiet \
--keyFile /opt/keyfile/mongodb-keyfile \
--replSet "rs0"
Step Nine

Get back on yourServerName1.yourDomain.com and let's add the other 2 servers to the replica set:

rs0:PRIMARY> rs.add("yourServerName2.yourDomain.com:27017")  
rs0:PRIMARY> rs.addArb("yourServerName3.yourDomain.com:30000")  

To confirm both servers have been added to our replica set, an output such as the one below has to be expected:

rs0:PRIMARY> rs.status()  
{
        "set" : "rs0",
        "date" : ISODate("2016-04-13T12:06:42.040Z"),
        "myState" : 1,
        "term" : NumberLong(37),
        "heartbeatIntervalMillis" : NumberLong(2000),
        "members" : [
                {
                        "_id" : 0,
                        "name" : "yourServerName1.yourDomain.com:27017",
                        "health" : 1,
                        "state" : 1,
                        "stateStr" : "PRIMARY",
                        "uptime" : 418664,
                        "optime" : {
                                "ts" : Timestamp(1460549202, 1),
                                "t" : NumberLong(37)
                        },
                        "optimeDate" : ISODate("2016-04-13T12:06:42Z"),
                        "electionTime" : Timestamp(1460132799, 1),
                        "electionDate" : ISODate("2016-04-08T16:26:39Z"),
                        "configVersion" : 180520,
                        "self" : true
                }
                {
                        "_id" : 1,
                        "name" : "yourServerName2.yourDomain.com:27017",
                        "health" : 1,
                        "state" : 2,
                        "stateStr" : "SECONDARY",
                        "uptime" : 185433,
                        "optime" : {
                                "ts" : Timestamp(1460549201, 6),
                                "t" : NumberLong(37)
                        },
                        "optimeDate" : ISODate("2016-04-13T12:06:41Z"),
                        "lastHeartbeat" : ISODate("2016-04-13T12:06:41.391Z"),
                        "lastHeartbeatRecv" : ISODate("2016-04-13T12:06:40.167Z"),
                        "pingMs" : NumberLong(0),
                        "syncingTo" : "yourServerName1.yourDomain.com:27017",
                        "configVersion" : 180520
                },
                {
                        "_id" : 2,
                        "name" : "yourServerName3.yourDomain.com:30000",
                        "health" : 1,
                        "state" : 7,
                        "stateStr" : "ARBITER",
                        "uptime" : 418654,
                        "lastHeartbeat" : ISODate("2016-04-13T12:06:41.990Z"),
                        "lastHeartbeatRecv" : ISODate("2016-04-13T12:06:40.808Z"),
                        "pingMs" : NumberLong(0),
                        "configVersion" : 180520
                }
        ],
        "ok" : 1
}

If rs.status() is ran right after adding the second server and the arbiter, one could notice stateStr value changing, reflecting the state of the server joining the replica set. You can read more about replica set members states one the official documentation.

Final notes

You can take a look at your mongoDB docker containers logs running on each server:

docker logs -ft mongo3.2.4  

There will be a time when the logs will be huge. Logs output can be shown from a given date and time, passing --since parameter:

docker logs -ft --since="2016-04-12T06:19:09.000000000Z" mongo3.2.4  

More information about docker logs and its options can be found here

Now that we have a mongoDB replica set cluster, we could, for example:

add a node:

rs.add("yourServerName4.yourDomain.com:27017")  

or remove a node:

rs.remove("yourServerName4.yourDomain.com:27017")  

Another useful command is:

rs0:PRIMARY> rs.stepDown()  

it will force the current primary to become secondary, triggering an election for primary. This is useful when you have to run maintenance tasks on that server.
You can read more about rs.stepDown() here

That's it for now. Enjoy your brand new mongoDB replica set docker cluster and put it to good use, inserting some data, and watch 'em spread on all of your servers.