Zookeeper Dynamic Config

In this post I'll describe how to use Zookeeper Dynamic Config, a feature introduced in version 3.5 of Zookeeper that allows you to dynamically add and remove nodes from an ensemble. Prior to the 3.5.0 release, the membership and all other configuration parameters of Zookeeper were static - loaded during boot and immutable at runtime. This resulted in operators having to execute rolling restarts to reconfigure clusters, sometimes resulting in data loss and inconsistency in production. All this will change as the 3.5 release reaches maturity ...

Prerequisites

Docker is not required to run Zookeeper, but I have created a Zookeeper Docker Image that makes it very easy to stop and start both single nodes and small ensembles. The rest of this blog post assumes that you have Docker installed.

Get an ensemble up and running

Start by downloading the docker-compose.yml file I've created from Github.

Run the following in a terminal:

$ docker-compose up

This will start a 3-node ensemble. Each node is given a unique hostname (zk-1, zk-2, ...) and port (starting at 2181, the default Zookeeper client port) and a persistent volume to store data. The table below summarises the configuration of the 3 nodes:

The docker-compose.yml file also creates a named bridge network, which allows containers to communicate with each other using the hostname (the default bridge network doesn't provide this feature as it breaks compatibility with older versions of Docker Compose).

The use of Docker Compose in this context is ideal for testing small clusters and confirming functionality - it is not, however, suitable for a production deployment as it gives you no resiliency - all the nodes are running on the same physical host and loss of that host will make your cluster unavailable.

Review ensemble configuration

Open another terminal and connect to this cluster:

$ zkCli.sh -server <docker.ip>

Where <docker.ip> represents your Docker host (if you're on MacOS X / Windows and using Docker Machine this will probably be 192.168.99.100 or similar, on Linux you shouldn't require the server argument at all as localhost is your Docker host).

This will connect to the default Zookeeper client port 2181 (i.e. the zookeeper_zk-1 container). You can also use <docker.ip>:2182 or <docker.ip>:2183 to connect to the zookeeper_zk-2 or zookeeper_zk-3 containers. When connecting to the ensemble as a client you would include all the servers in your connection string to ensure your code is resilient when one or more of the nodes is unavailable.

You can review the current configuration by running the following command inside the Zookeeper shell:

[zk: 192.168.99.100(CONNECTED) 1] get /zookeeper/config

server.1=zk-1:2888:3888:participant;0.0.0.0:2181
server.2=zk-2:2888:3888:participant;0.0.0.0:2181
server.3=zk-3:2888:3888:participant;0.0.0.0:2181
version=300000000

How to dynamically add a node

Now lets use the new dynamic config features. Firstly start up a new node that knows about the 3 others - my docker image makes this easy by creating the configuration files dynamically for you. To start a node, zk-4, that is aware of the existing ensemble:

$ docker run -it --net=zookeeper_zk \
  -p 2184:2181 \
  -v zookeeper_zookeeper-data4:/var/opt/zookeeper \
  -h zk-4 \
  mrhornsby/zookeeper:3.5.1-alpha \
  4 1 2 3

Here's an explanation of the various components of the command:

• --net=zookeeper_zk - Add this container to the zookeeper_zk network so that it can see the other containers
• -p 2184:1281 - Expose port 2181 of the container as port 2184 on the Docker host (2181, 2182 and 2183 are already taken by the existing containers)
• -v zookeeper_zookeeper-data4:/var/opt/zookeeper - Create a data volume called zookeeper_zookeeper-data4 and mount it at /var/opt/zookeeper in the container
• -h zk-4 - Give this container hostname zk-4
• mrhornsby/zookeeper:3.5.1-alpha - My Zookeeper Docker Image on Docker Hub
• 4 1 2 3 - Arguments to the container. My node id is 4 and there are other nodes 1 2 3

You now have a 3-node zookeeper ensemble with a 4th node as an observer, but the original 3 are not yet aware of the new node. Return to the Zookeeper shell and run the following command:

[zk: 192.168.99.100(CONNECTED) 2] reconfig -add server.4=zk-4:2888:3888:participant;0.0.0.0:2181

Committed new configuration:
server.1=zk-1:2888:3888:participant;0.0.0.0:2181
server.2=zk-2:2888:3888:participant;0.0.0.0:2181
server.3=zk-3:2888:3888:participant;0.0.0.0:2181
server.4=zk-4:2888:3888:participant;0.0.0.0:2181
version=300000002

Now the ensemble is reconfigured with 4 nodes.

You can verify this by reviewing the logs of the containers - you will see messages similar to the below in the log of the leader:

zk-3_1  | 2016-07-11 17:00:41,756 [myid:3] - INFO  [LearnerHandler-/172.19.0.5:43330:LearnerHandler@683] - Synchronizing with Follower sid: 4 maxCommittedLog=0x300000002 minCommittedLog=0x200000001 lastProcessedZxid=0x300000002 peerLastZxid=0x300000001

How to dynamically remove a node

Now let's go about removing a node dynamically. Firstly we need to reconfig as we did above, but this time using the -remove command. Go to your zkCli.sh shell and execute the following:

[zk: 192.168.99.100:2182(CONNECTED) 3] reconfig -remove 3

Committed new configuration:
server.1=zk-1:2888:3888:participant;0.0.0.0:2181
server.2=zk-2:2888:3888:participant;0.0.0.0:2181
server.4=zk-4:2888:3888:participant;0.0.0.0:2181
version=300000009

You may see some exceptions as nodes disconnect from each other and possibly a new leader election.

Be aware that above I've removed a node that my client is not connected to. If you choose to remove the node that you connected to with zkCli.sh then you may see different behaviour to the above!

Although the ensemble is now running with 3 nodes you still need to shutdown the zookeeper process. Do this by running the following:

$ docker stop zookeeper_zk-3_1

$ docker rm zookeeper_zk-3_1

You will see the node terminate and probably various chatter from the other nodes, but you can confirm you have a fully functioning ensemble by connection to any of the reminaing nodes, zk-1, zk-2 and zk-4.

Summary

Today you've walked through using Docker Compose to generate a small Zookeeper test ensemble, verified that it is up and running and then dynamically added a new node and then removed a pre-existing node.

I hope you found the tutorial useful, feel free to drop me a mail with any questions.