1. Apache ZooKeeper By Patrick Hunt, Mahadev Konar

2. What is ZooKeeper?
ZooKeeper is much more than a distributed lock server! A highly available, scalable, distributed, configuration, consensus, group membership, leader election, naming, and coordination service

3. Why use ZooKeeper?
Difficulty of implementing these kinds of services reliably
brittle in the presence of change
difficult to manage
different implementations lead to management complexity when the applications are deployed

4. What is ZooKeeper again?
File api without partial reads/writes
No renames
Ordered updates and strong persistence guarantees
Conditional updates (version)
Watches for data changes
Ephemeral nodes
Generated file names

5. Any Guarantees? Clients will never detect old data.
Clients will get notified of a change to data they are watching within a bounded period of time.
All requests from a client will be processed in order.
All results received by a client will be consistent with results received by all other clients.

6. Data Model Hierarchal namespace Each znode has data and children
data is read and written in its entirety /
services
YaView
servers
stupidname
morestupidity
locks
read-1
apps
users

7. ZooKeeper API String create(path, data, acl, flags)‏
void delete(path, expectedVersion)‏
Stat setData(path, data, expectedVersion)‏
(data, Stat) getData(path, watch)‏
Stat exists(path, watch)‏
String[] getChildren(path, watch)‏
void sync(path)‏

8. ZooKeeper Service ZooKeeper Service Server Server Server Server Server
Leader
Server
Server
Server
Server
Server
Server
Client
Client
Client
Client
Client
Client
Client
All servers store a copy of the data (in memory)‏
A leader is elected at startup
Followers service clients, all updates go through leader
Update responses are sent when a majority of servers have persisted the change

9. Use cases inside of Yahoo!
Leader Election
Group Membership
Work Queues
Configuration Management
Cluster Management
Load Balancing
Sharding

10. Leader Election getdata(“/servers/leader”, true)
if successful follow the leader described in the data
and exit
3 create(“/servers/leader”, hostname, EPHEMERAL)
if successful lead and exit
goto step 1

11. Leader Election in Python
handle = zookeeper.init("localhost:2181", my_connection_watcher, 10000, 0) (data, stat) = zookeeper.get(handle, “/app/leader”, True); if (stat == None) path = zookeeper.create(handle, “/app/leader”, hostname:info, [ZOO_OPEN_ACL_UNSAFE], zookeeper.EPHEMERAL) if (path == None) (data, stat) = zookeeper.get(handle, “/app/leader”, True) #someone else is the leader # parse the string path that contains the leader address else # we are the leader continue leading #someone else is the leader #parse the string path that contains the leader address

12. Cluster Management /nodes node-1 node-2 node-3 Monitoring process:
Watch on /nodes
On watch trigger do
getChildren(/nodes, true)
Track which nodes have
gone away
Each Node:
Create /nodes/node-${i} as ephemeral nodes
Keep updating /nodes/node-${i} periodically for node status changes (status updates could be load/iostat/cpu/others)
/nodes
node-1
node-2
node-3

13. Work Queues /tasks task-1 task-2 task-3 /machines m-1 task-1
Monitoring process:
Watch /tasks for published
tasks
Pick tasks on watch trigger from
/tasks
3. assign it to a machine specific
queue by creating
create(/machines/m-${i}/task-${j})
Watch for deletion of tasks (maps to task
completion)
Machine process:
Machines watch for /(/machines/m-${i})
for any creation of tasks
2. After executing task-${i} delete task-${i} from
/tasks and /m-${i}
/tasks
task-1
task-2
task-3
/machines
m-1
task-1

14. Performance Numbers.

15. Monitoring Tools
Command port
JMX
Trace Logging

16. Where are we? Multi Tenant Observers Recipes Bindings
Reusuable code libraries
Bindings
Java, C, Perl, Python, REST, Ruby?

17. Where are we? BookKeeper (a contrib project)
System to reliably log streams of records
Using BookKeeper and ZooKeeper for a pub sub system

18. Who is using ZooKeeper? Hbase Solr Digg LinkedIn
Unnamed financial institutions

19. What do we do next? WAN – more testing
Cross colo quorum
Client server in different colo’s
Usability – timeouts from zookeeper clients are a headache – ZOOKEEPER-22

20. Q&A
Questions?
Links: