Presentation is loading. Please wait.

Presentation is loading. Please wait.

Distributed Cluster Repair for OceanStore Irena Nadjakova and Arindam Chakrabarti Acknowledgements: Hakim Weatherspoon John Kubiatowicz.

Published by Modified over 9 years ago

Similar presentations

Presentation on theme: "Distributed Cluster Repair for OceanStore Irena Nadjakova and Arindam Chakrabarti Acknowledgements: Hakim Weatherspoon John Kubiatowicz."— Presentation transcript:

1 Distributed Cluster Repair for OceanStore Irena Nadjakova and Arindam Chakrabarti Acknowledgements: Hakim Weatherspoon John Kubiatowicz

2 Dec 9, 2003Distributed Cluster Repair for OceanStore2 OceanStore Overview Data Storage Utility Robustness Security Durability High availability Global-scale

3 Dec 9, 2003Distributed Cluster Repair for OceanStore3 Where our project fits in Durability Automatic version-management Highly redundant erasure-coding Massive dissemination of fragments on machines with highly uncorrelated availability.

4 Dec 9, 2003Distributed Cluster Repair for OceanStore4 The internet

5 Dec 9, 2003Distributed Cluster Repair for OceanStore5 Choosing locations for storing a fragment

6 Dec 9, 2003Distributed Cluster Repair for OceanStore6 Choosing locations for storing a fragment

7 Dec 9, 2003Distributed Cluster Repair for OceanStore7 Choosing locations for storing a fragment

8 Dec 9, 2003Distributed Cluster Repair for OceanStore8 Clustering

9 Dec 9, 2003Distributed Cluster Repair for OceanStore9 Clustering

10 Dec 9, 2003Distributed Cluster Repair for OceanStore10 OceanStore solution Availability of each machine tracked over time. Machines that have very little availability are not used for fragment storage. Distance between each pair of machines computed. (high mutual information ) close) Cluster the machines into chunks based on this distance using normalized cuts. All the computation is done on one central computer (Cluster Server).

11 Dec 9, 2003Distributed Cluster Repair for OceanStore11 OceanStore solution Machines that are highly correlated in availability are in same cluster. Machines in separate clusters have low correlation in availability. When a node needs to store replica fragments, it requests cluster information from the cluster server and uses it to send each fragment to k nodes: one from each of k different clusters.

12 Dec 9, 2003Distributed Cluster Repair for OceanStore12 Cluster creation Needs centralized computation. Can we do it in a distributed manner ? NCuts is one stumbling block. It seems to need the entire graph. Having to pull the cluster info from one central cluster server: single point of failure Can we have a “Distributed NCuts” algo to look at subgraphs ? How to make subgraphs? Do we need to know the entire graph to decide how to divide it into pieces ?

13 Dec 9, 2003Distributed Cluster Repair for OceanStore13 Distributed clustering

14 Dec 9, 2003Distributed Cluster Repair for OceanStore14 Distributed clustering

15 Dec 9, 2003Distributed Cluster Repair for OceanStore15 Distributed clustering

16 Dec 9, 2003Distributed Cluster Repair for OceanStore16 Distributed clustering

17 Dec 9, 2003Distributed Cluster Repair for OceanStore17 Distributed clustering

18 Dec 9, 2003Distributed Cluster Repair for OceanStore18 Initial idea We run the centralized algorithm once for some time period (chose 73 days) to generate some initial clustering (expensive!) We distribute the machines among some f cluster servers –Each has a smaller subset of size num of the initial machines –Keeping the initial clustering proportions for each node –Each machine occurs in approximately equal number of cluster servers

19 Dec 9, 2003Distributed Cluster Repair for OceanStore19 Initial idea (cont) Now we can afford to recluster the machines on each server frequently to keep up with the network changes. –Chose to do it once every 30 days for the simulation purposes, but can easily be done a lot more often

20 Dec 9, 2003Distributed Cluster Repair for OceanStore20 Evaluation To see how well this does, we want to compare it with the original global algorithm, run in the same time period. Metric – the average mutual information I(x,y) =  P(x,y) log P(x,y)/P(x)P(y) –Average MI for a single server is just the average of the mutual information between pairs of machines in different clusters on the server –On multiple servers, we compute the above on every server, then average among servers

21 Dec 9, 2003Distributed Cluster Repair for OceanStore21 Simulating Network Evolution Dynamics We have availability data for 1000 machines for a period of 73 days. We use it to simulate the behavior of a network with 1000 machines over a period of 730 days = 2 years. We simulate networks with varying evolution characteristics to evaluate the robustness of our distributed cluster repair algorithm.

22 Dec 9, 2003Distributed Cluster Repair for OceanStore22 Simulating Network Evolution Dynamics Qualities of a good network: Maybe server availability (AV) should not vary drastically in the future ? Maybe average server repair time (MTTR) should not vary drastically ? Maybe mean time to failure (MTTF) should not vary drastically ? Maybe failure correlations (FCOR) should also not vary drastically ?

23 Dec 9, 2003Distributed Cluster Repair for OceanStore23 NS Algo 1: Sanity Check 1 Global déjà vu Maintains AV, MTTF, MTTR, FCOR Simulates a well-behaved network. Our distributed update algorithm should do very well on this.

24 Dec 9, 2003Distributed Cluster Repair for OceanStore24 NS Algo 2: Acid Test 1 Local déjà vu Maintains AV, MTTF, MTTR, but not FCOR.

25 Dec 9, 2003Distributed Cluster Repair for OceanStore25 NS Algo 3: Acid Test 2 Births and Deaths Maintains AV, MTTF, MTTR, and FCOR, but only for some nodes, and for some time. Nodes are taken off (die) the network or are added to (born) the network at certain times. When they are actually on the network, they maintain their AV, MTTF, MTTR, FCOR.

26 Dec 9, 2003Distributed Cluster Repair for OceanStore26 NS Algo 4: Acid Test 3 Noisy Global déjà vu Maintains AV, MTTF, MTTR, FCOR to a large extent, but adds some Gaussian noise, representing the variations that may be observed in a real network.

27 Dec 9, 2003Distributed Cluster Repair for OceanStore27 NS Algo 5: Acid Test 4 Noisy Local déjà vu Maintains AV, MTTF, MTTR, but not FCOR, and also adds some Gaussian noise representing the variations that may be observed in a real network. Does our algorithm do well in this situation ? If yes, how robust is it to noise ?

28 Dec 9, 2003Distributed Cluster Repair for OceanStore28

29 Dec 9, 2003Distributed Cluster Repair for OceanStore29

30 Dec 9, 2003Distributed Cluster Repair for OceanStore30

31 Dec 9, 2003Distributed Cluster Repair for OceanStore31

32 Dec 9, 2003Distributed Cluster Repair for OceanStore32 Still problems Initial clustering is expensive What happens if we don’t use it?

33 Dec 9, 2003Distributed Cluster Repair for OceanStore33

34 Dec 9, 2003Distributed Cluster Repair for OceanStore34 How to fix this? Randomly distribute machines to the servers Perform local clustering Find the ‘unwanted’ elements (highest mutual information with the rest on this node) Exchange them with ‘unwanted’ elements of another cluster to which the first ones are least correlated Communication overhead is low; most computation can proceed without a lot of communication

35 Dec 9, 2003Distributed Cluster Repair for OceanStore35 Under development… What we have so far is a scheme that –picks a server at random –finds a few unwanted elements –exchanges those with the same number of unwanted elements of another server – picked at random, or having the best correlation with the unwanted elements of the first server The percentage improvement is small so far – 0.4%-1.5% for the first 5 or so runs. It falls off afterwards.

36 Dec 9, 2003Distributed Cluster Repair for OceanStore36 How to improve Exchange more than 1 machines ? Run for several generations ? It may be even better to just randomly exchange machines, as long as the overall average mutual information of the distributed cluster decreases.

37 Dec 9, 2003Distributed Cluster Repair for OceanStore37 Summary of achievements Towards getting rid of expensive centralized cluster creation Scalable distributed cluster management scheme

38 Dec 9, 2003Distributed Cluster Repair for OceanStore38 Thanks for listening ! Acknowledgements: Hakim Weatherspoon John Kubiatowicz

Download ppt "Distributed Cluster Repair for OceanStore Irena Nadjakova and Arindam Chakrabarti Acknowledgements: Hakim Weatherspoon John Kubiatowicz."

Similar presentations

Background Virtual memory – separation of user logical memory from physical memory. Only part of the program needs to be in memory for execution. Logical.

Background Virtual memory – separation of user logical memory from physical memory. Only part of the program needs to be in memory for execution. Logical.
CS3771 Today: deadlock detection and election algorithms  Previous class Event ordering in distributed systems Various approaches for Mutual Exclusion.

CS3771 Today: deadlock detection and election algorithms  Previous class Event ordering in distributed systems Various approaches for Mutual Exclusion.
Imbalanced data David Kauchak CS 451 – Fall 2013.

Imbalanced data David Kauchak CS 451 – Fall 2013.
1 Greedy Forwarding in Dynamic Scale-Free Networks Embedded in Hyperbolic Metric Spaces Dmitri Krioukov CAIDA/UCSD Joint work with F. Papadopoulos, M.

1 Greedy Forwarding in Dynamic Scale-Free Networks Embedded in Hyperbolic Metric Spaces Dmitri Krioukov CAIDA/UCSD Joint work with F. Papadopoulos, M.
1 Accessing nearby copies of replicated objects Greg Plaxton, Rajmohan Rajaraman, Andrea Richa SPAA 1997.

1 Accessing nearby copies of replicated objects Greg Plaxton, Rajmohan Rajaraman, Andrea Richa SPAA 1997.
Scalable Content-aware Request Distribution in Cluster-based Network Servers Jianbin Wei 10/4/2001.

Scalable Content-aware Request Distribution in Cluster-based Network Servers Jianbin Wei 10/4/2001.
Directional triadic closure and edge deletion mechanism induce asymmetry in directed edge properties.

Directional triadic closure and edge deletion mechanism induce asymmetry in directed edge properties.
Small-world Overlay P2P Network

Small-world Overlay P2P Network
Scaling Distributed Machine Learning with the BASED ON THE PAPER AND PRESENTATION: SCALING DISTRIBUTED MACHINE LEARNING WITH THE PARAMETER SERVER – GOOGLE,

Scaling Distributed Machine Learning with the BASED ON THE PAPER AND PRESENTATION: SCALING DISTRIBUTED MACHINE LEARNING WITH THE PARAMETER SERVER – GOOGLE,
CS 582 / CMPE 481 Distributed Systems

CS 582 / CMPE 481 Distributed Systems
1 Data Persistence in Large-scale Sensor Networks with Decentralized Fountain Codes Yunfeng Lin, Ben Liang, Baochun Li INFOCOM 2007.

1 Data Persistence in Large-scale Sensor Networks with Decentralized Fountain Codes Yunfeng Lin, Ben Liang, Baochun Li INFOCOM 2007.
G22.3250-001 Robert Grimm New York University Sprite LFS or Let’s Log Everything.

G Robert Grimm New York University Sprite LFS or Let’s Log Everything.
Availability in Global Peer-to-Peer Systems Qin (Chris) Xin, Ethan L. Miller Storage Systems Research Center University of California, Santa Cruz Thomas.

Availability in Global Peer-to-Peer Systems Qin (Chris) Xin, Ethan L. Miller Storage Systems Research Center University of California, Santa Cruz Thomas.
© University of Minnesota Data Mining for the Discovery of Ocean Climate Indices 1 CSci 8980: Data Mining (Fall 2002) Vipin Kumar Army High Performance.

© University of Minnesota Data Mining for the Discovery of Ocean Climate Indices 1 CSci 8980: Data Mining (Fall 2002) Vipin Kumar Army High Performance.
HCS Clustering Algorithm

HCS Clustering Algorithm
Goal: To build a ubiquitous and robust storage infrastructure Requirement: Scalability, availability, performance, robustness Solution: Dynamic object.

Goal: To build a ubiquitous and robust storage infrastructure Requirement: Scalability, availability, performance, robustness Solution: Dynamic object.
Dept. of Computer Science & Engineering, CUHK1 Trust- and Clustering-Based Authentication Services in Mobile Ad Hoc Networks Edith Ngai and Michael R.

Dept. of Computer Science & Engineering, CUHK1 Trust- and Clustering-Based Authentication Services in Mobile Ad Hoc Networks Edith Ngai and Michael R.
Efficient and Robust Computation of Resource Clusters in the Internet Efficient and Robust Computation of Resource Clusters in the Internet Chuang Liu,

Efficient and Robust Computation of Resource Clusters in the Internet Efficient and Robust Computation of Resource Clusters in the Internet Chuang Liu,
Efficient replica maintenance for distributed storage systems Byung-Gon Chun, Frank Dabek, Andreas Haeberlen, Emil Sit, Hakim Weatherspoon, M. Frans Kaashoek,

Efficient replica maintenance for distributed storage systems Byung-Gon Chun, Frank Dabek, Andreas Haeberlen, Emil Sit, Hakim Weatherspoon, M. Frans Kaashoek,
Fault-tolerant Adaptive Divisible Load Scheduling Xuan Lin, Sumanth J. V. Acknowledge: a few slides of DLT are from Thomas Robertazzi ’ s presentation.

Fault-tolerant Adaptive Divisible Load Scheduling Xuan Lin, Sumanth J. V. Acknowledge: a few slides of DLT are from Thomas Robertazzi ’ s presentation.

Similar presentations

Ads by Google