Building a Kubernetes PetSet Cluster for your MongoDB ReplicaSet
Some observations when working with Kubernetes 1.3 introduced PetSets to build MongoDB.
Here’s my PetSet YAML:
# mypetset.yml
apiVersion: v1
kind: Service
metadata:
annotations:
service.alpha.kubernetes.io/tolerate-unready-endpoints: "true"
name: a-mongo
labels:
name: a-mongo
tier: database
type: mongo
spec:
ports:
- port: 27017
name: mongo
type: NodePort
selector:
tier: database
name: a-mongo
---
apiVersion: apps/v1alpha1
kind: PetSet
metadata:
name: a-mongo
spec:
serviceName: "a-mongo"
replicas: 3
template:
metadata:
labels:
name: a-mongo
tier: database
annotations:
pod.alpha.kubernetes.io/initialized: "true"
spec:
terminationGracePeriodSeconds: 10
containers:
- name: mongo
image: mongo:3.0.12
command:
- mongod
- "--replSet"
- rs0
imagePullPolicy: Always
ports:
- containerPort: 27017
name: mongo
resources:
requests:
cpu: "0.1m"
memory: "64Mi"
limits:
cpu: "1"
memory: "1Gi"
volumeMounts:
- name: data
mountPath: /data/db
volumeClaimTemplates:
- metadata:
name: data
annotations:
volume.alpha.kubernetes.io/storage-class: fo
spec:
accessModes: [ "ReadWriteOnce" ]
resources:
requests:
storage: 10Gi
If you’re on AWS, or GCE, your mongo instances are backed by persistent storage (ElasticBlockStore, GCEDisk) that will ensure that even if one of your pets die, when its revived, it will reuse the same storage.
So with your config_file, lets start initializing your petset.
$ kubectl create -f mypetset.yml
Give it a while and you should see 3 pods appear.
Note: In PetSets, pods are started one after another so if one of your pets fails startup, any other pets in the order that hasn’t been started will not be started.
$ kubectl get pods
a-mongo-0 1/1 Running 0 22m
a-mongo-1 1/1 Running 0 21m
a-mongo-2 1/1 Running 0 21m
Ok looks like we’re done! Let’s initiate the cluster.
First, lets get all the available IP of our cluster.
$ kubectl describe pods a-mongo | grep IP
IP: 10.244.6.3
IP: 10.244.7.3
IP: 10.244.6.4
Awesome. Now keep this somewhere handy.
$ kubectl exec -it a-mongo-0 /bin/bash
root@a-mongo-0 $ mongo
> rs.initiate()
One of the biggest caveat in setting up your mongo replicaset on PetSets is that you need to perform an extra step before adding your other cluster. We want to do this before we add members so that we don’t run into an issue where we lose our PRIMARY node due to configuration.
> cfg = rs.conf()
> cfg.members[0].host = "10.244.6.3:27017" # replace this with the cluster IP of your a-mongo-0 container.
> rs.reconfig(cfg)
Now you can safely add your members.
Note: So what happened? Apparently MongoDB ReplicaSets do not play nice if one of its members is inconsistent in its naming. ie. If you’re using IP addresses, all your cluster members should be using it. Reference
rs0:PRIMARY> rs.add("10.244.7.3")
rs0:PRIMARY> rs.add("10.244.6.4")
If you do a rs.status(), you should see the other 2 members listed as SECONDARY.
And now you have a full cluster!