Presentation is loading. Please wait.

Presentation is loading. Please wait.

Apache ZooKeeper By Patrick Hunt, Mahadev Konar

Similar presentations

Presentation on theme: "Apache ZooKeeper By Patrick Hunt, Mahadev Konar"— Presentation transcript:

1 Apache ZooKeeper By Patrick Hunt, Mahadev Konar

2 What is ZooKeeper? ZooKeeper is much more than a distributed lock server! A highly available, scalable, distributed, configuration, consensus, group membership, leader election, naming, and coordination service

3 Why use ZooKeeper? Difficulty of implementing these kinds of services reliably brittle in the presence of change difficult to manage different implementations lead to management complexity when the applications are deployed

4 What is ZooKeeper again?
File api without partial reads/writes No renames Ordered updates and strong persistence guarantees Conditional updates (version) Watches for data changes Ephemeral nodes Generated file names

5 Any Guarantees? Clients will never detect old data.
Clients will get notified of a change to data they are watching within a bounded period of time. All requests from a client will be processed in order. All results received by a client will be consistent with results received by all other clients.

6 Data Model Hierarchal namespace Each znode has data and children
data is read and written in its entirety / services YaView servers stupidname morestupidity locks read-1 apps users

7 ZooKeeper API String create(path, data, acl, flags)‏
void delete(path, expectedVersion)‏ Stat setData(path, data, expectedVersion)‏ (data, Stat) getData(path, watch)‏ Stat exists(path, watch)‏ String[] getChildren(path, watch)‏ void sync(path)‏

8 ZooKeeper Service ZooKeeper Service Server Server Server Server Server
Leader Server Server Server Server Server Server Client Client Client Client Client Client Client All servers store a copy of the data (in memory)‏ A leader is elected at startup Followers service clients, all updates go through leader Update responses are sent when a majority of servers have persisted the change

9 Use cases inside of Yahoo!
Leader Election Group Membership Work Queues Configuration Management Cluster Management Load Balancing Sharding

10 Leader Election getdata(“/servers/leader”, true)
if successful follow the leader described in the data and exit 3 create(“/servers/leader”, hostname, EPHEMERAL) if successful lead and exit goto step 1

11 Leader Election in Python
handle = zookeeper.init("localhost:2181", my_connection_watcher, 10000, 0) (data, stat) = zookeeper.get(handle, “/app/leader”, True); if (stat == None) path = zookeeper.create(handle, “/app/leader”, hostname:info, [ZOO_OPEN_ACL_UNSAFE], zookeeper.EPHEMERAL) if (path == None) (data, stat) = zookeeper.get(handle, “/app/leader”, True) #someone else is the leader # parse the string path that contains the leader address else # we are the leader continue leading #someone else is the leader #parse the string path that contains the leader address

12 Cluster Management /nodes node-1 node-2 node-3 Monitoring process:
Watch on /nodes On watch trigger do getChildren(/nodes, true) Track which nodes have gone away Each Node: Create /nodes/node-${i} as ephemeral nodes Keep updating /nodes/node-${i} periodically for node status changes (status updates could be load/iostat/cpu/others) /nodes node-1 node-2 node-3

13 Work Queues /tasks task-1 task-2 task-3 /machines m-1 task-1
Monitoring process: Watch /tasks for published tasks Pick tasks on watch trigger from /tasks 3. assign it to a machine specific queue by creating create(/machines/m-${i}/task-${j}) Watch for deletion of tasks (maps to task completion) Machine process: Machines watch for /(/machines/m-${i}) for any creation of tasks 2. After executing task-${i} delete task-${i} from /tasks and /m-${i} /tasks task-1 task-2 task-3 /machines m-1 task-1

14 Performance Numbers.

15 Monitoring Tools Command port JMX Trace Logging

16 Where are we? Multi Tenant Observers Recipes Bindings
Reusuable code libraries Bindings Java, C, Perl, Python, REST, Ruby?

17 Where are we? BookKeeper (a contrib project)
System to reliably log streams of records Using BookKeeper and ZooKeeper for a pub sub system

18 Who is using ZooKeeper? Hbase Solr Digg LinkedIn
Unnamed financial institutions

19 What do we do next? WAN – more testing
Cross colo quorum Client server in different colo’s Usability – timeouts from zookeeper clients are a headache – ZOOKEEPER-22

20 Q&A Questions? Links:

Download ppt "Apache ZooKeeper By Patrick Hunt, Mahadev Konar"

Similar presentations

Ads by Google