Zookeeper Dynamic Config
Posted by Mark Hornsby on 2016-07-29
In this post I’ll describe how to use Zookeeper Dynamic Config, a feature introduced in version 3.5 of Zookeeper that allows you to dynamically add and remove nodes from an ensemble. Prior to the 3.5.0 release, the membership and all other configuration parameters of Zookeeper were static - loaded during boot and immutable at runtime. This resulted in operators having to execute rolling restarts to reconfigure clusters, sometimes resulting in data loss and inconsistency in production. All this will change as the 3.5 release reaches maturity …
Docker is not required to run Zookeeper, but I have created a Zookeeper Docker Image that makes it very easy to stop and start both single nodes and small ensembles. The rest of this blog post assumes that you have Docker installed.
Get an ensemble up and running
Start by downloading the docker-compose.yml file I’ve created from Github.
Run the following in a terminal:
$ docker-compose up
This will start a 3-node ensemble. Each node is given a unique hostname (zk-1, zk-2, …) and port (starting at 2181, the default Zookeeper client port) and a persistent volume to store data. The table below summarises the configuration of the 3 nodes:
|Container name||Hostname||Port||Volume name|
The docker-compose.yml file also creates a named bridge network, which allows containers to communicate with each other using the hostname (the default bridge network doesn’t provide this feature as it breaks compatibility with older versions of Docker Compose).
The use of Docker Compose in this context is ideal for testing small clusters and confirming functionality - it is not, however, suitable for a production deployment as it gives you no resiliency - all the nodes are running on the same physical host and loss of that host will make your cluster unavailable.
Review ensemble configuration
Open another terminal and connect to this cluster:
$ zkCli.sh -server <docker.ip>
Where <docker.ip> represents your Docker host (if you’re on MacOS X / Windows and using Docker Machine this will probably be 192.168.99.100 or similar, on Linux you shouldn’t require the server argument at all as localhost is your Docker host).
This will connect to the default Zookeeper client port 2181 (i.e. the zookeeper_zk-1 container). You can also use <docker.ip>:2182 or <docker.ip>:2183 to connect to the zookeeper_zk-2 or zookeeper_zk-3 containers. When connecting to the ensemble as a client you would include all the servers in your connection string to ensure your code is resilient when one or more of the nodes is unavailable.
You can review the current configuration by running the following command inside the Zookeeper shell:
[zk: 192.168.99.100(CONNECTED) 1] get /zookeeper/config
server.1=zk-1:2888:3888:participant;0.0.0.0:2181 server.2=zk-2:2888:3888:participant;0.0.0.0:2181 server.3=zk-3:2888:3888:participant;0.0.0.0:2181 version=300000000
How to dynamically add a node
Now lets use the new dynamic config features. Firstly start up a new node that knows about the 3 others - my docker image makes this easy by creating the configuration files dynamically for you. To start a node, zk-4, that is aware of the existing ensemble:
$ docker run -it --net=zookeeper_zk \ -p 2184:2181 \ -v zookeeper_zookeeper-data4:/var/opt/zookeeper \ -h zk-4 \ mrhornsby/zookeeper:3.5.1-alpha \ 4 1 2 3
Here’s an explanation of the various components of the command:
- –net=zookeeper_zk - Add this container to the zookeeper_zk network so that it can see the other containers
- -p 2184:1281 - Expose port 2181 of the container as port 2184 on the Docker host (2181, 2182 and 2183 are already taken by the existing containers)
- -v zookeeper_zookeeper-data4:/var/opt/zookeeper - Create a data volume called zookeeper_zookeeper-data4 and mount it at /var/opt/zookeeper in the container
- -h zk-4 - Give this container hostname zk-4
- mrhornsby/zookeeper:3.5.1-alpha - My Zookeeper Docker Image on Docker Hub
- 4 1 2 3 - Arguments to the container. My node id is 4 and there are other nodes 1 2 3
You now have a 3-node zookeeper ensemble with a 4th node as an observer, but the original 3 are not yet aware of the new node. Return to the Zookeeper shell and run the following command:
[zk: 192.168.99.100(CONNECTED) 2] reconfig -add server.4=zk-4:2888:3888:participant;0.0.0.0:2181
Committed new configuration: server.1=zk-1:2888:3888:participant;0.0.0.0:2181 server.2=zk-2:2888:3888:participant;0.0.0.0:2181 server.3=zk-3:2888:3888:participant;0.0.0.0:2181 server.4=zk-4:2888:3888:participant;0.0.0.0:2181 version=300000002
Now the ensemble is reconfigured with 4 nodes.
You can verify this by reviewing the logs of the containers - you will see messages similar to the below in the log of the leader:
zk-3_1 | 2016-07-11 17:00:41,756 [myid:3] - INFO [LearnerHandler-/172.19.0.5:43330:[email protected]] - Synchronizing with Follower sid: 4 maxCommittedLog=0x300000002 minCommittedLog=0x200000001 lastProcessedZxid=0x300000002 peerLastZxid=0x300000001
How to dynamically remove a node
Now let’s go about removing a node dynamically. Firstly we need to reconfig as we did above, but this time using the -remove command. Go to your zkCli.sh shell and execute the following:
[zk: 192.168.99.100:2182(CONNECTED) 3] reconfig -remove 3
Committed new configuration: server.1=zk-1:2888:3888:participant;0.0.0.0:2181 server.2=zk-2:2888:3888:participant;0.0.0.0:2181 server.4=zk-4:2888:3888:participant;0.0.0.0:2181 version=300000009
You may see some exceptions as nodes disconnect from each other and possibly a new leader election.
Be aware that above I’ve removed a node that my client is not connected to. If you choose to remove the node that you connected to with zkCli.sh then you may see different behaviour to the above!
Although the ensemble is now running with 3 nodes you still need to shutdown the zookeeper process. Do this by running the following:
$ docker stop zookeeper_zk-3_1
$ docker rm zookeeper_zk-3_1
You will see the node terminate and probably various chatter from the other nodes, but you can confirm you have a fully functioning ensemble by connection to any of the reminaing nodes, zk-1, zk-2 and zk-4.
Today you’ve walked through using Docker Compose to generate a small Zookeeper test ensemble, verified that it is up and running and then dynamically added a new node and then removed a pre-existing node.
I hope you found the tutorial useful, feel free to drop me a mail with any questions.