1. November 17, 2015Department of Computer Sciences, UT Austin1 SDIMS: A Scalable Distributed Information Management System Praveen Yalagandula Mike Dahlin Laboratory for Advanced Systems Research (LASR) University of Texas at Austin

2. November 17, 2015Department of Computer Sciences, UT Austin2 Goal  A Distributed Operating System Backbone  Information collection and management  Core functionality of large distributed systems  Monitor, query and react to changes in the system  Examples:  Benefits  Ease development of new services  Facilitate deployment  Avoid repetition of same task by different services  Optimize system performance System administration and management Service placement and location Sensor monitoring and control Distributed Denial-of-Service attack detection File location service Multicast tree construction Naming and request routing …………

3. November 17, 2015Department of Computer Sciences, UT Austin3 Contributions – SDIMS  Provides an important basic building block  Information collection and management  Satisfies key requirements  Scalability With both nodes and attributes Leverage Distributed Hash Tables (DHT)  Flexibility Enable applications to control the aggregation Provide flexible API: install, update and probe  Autonomy Enable administrators to control flow of information Build Autonomous DHTs  Robustness Handle failures gracefully Perform re-aggregation upon failures –Lazy (by default) and On-demand (optional)

4. November 17, 2015Department of Computer Sciences, UT Austin4 Outline  SDIMS: a distributed operating system backbone  Aggregation abstraction  Our approach  Design  Prototype  Simulation and experimental results  Conclusions

5. November 17, 2015Department of Computer Sciences, UT Austin5 Aggregation Abstraction  Attributes  Information at machines  Aggregation tree  Physical nodes are leaves  Each virtual node represents a logical group of nodes Administrative domains, groups within domains, etc.  Aggregation function, f, for attribute A  Computes the aggregated value A i for level-i subtree A 0 = locally stored value at the physical node or NULL A i = f(A i-1 0, A i-1 1, …, A i-1 k ) for virtual node with k children  Each virtual node is simulated by one or more machines  Example: Total users logged in the system  Attribute: numUsers  Aggregation Function: Summation a b c d A0A0 A1A1 A2A2 f(a,b) f(c,d) f(f(a,b), f(c,d)) 4 1 2 2 5 4 9 Astrolabe [VanRenesse et al TOCS’03]

6. November 17, 2015Department of Computer Sciences, UT Austin6 Outline  SDIMS: a distributed operating system backbone  Aggregation abstraction  Our approach  Design  Prototype  Simulation and experimental results  Conclusions

7. November 17, 2015Department of Computer Sciences, UT Austin7 Scalability with nodes and attributes  To be a basic building block, SDIMS should support  Large number of machines Enterprise and global-scale services Trend: Large number of small devices  Applications with a large number of attributes Example: File location system –Each file is an attribute –Large number of attributes  Challenges: build aggregation trees in a scalable way  Build multiple trees Single tree for all attributes  load imbalance  Ensure small number of children per node in the tree Reduces maximum node stress

8. November 17, 2015Department of Computer Sciences, UT Austin8 Building aggregation trees: Exploit DHTs  A DHT can be viewed as multiple aggregation trees  Distributed Hash Tables (DHTs)  Each node assigned an ID from large space (160-bit)  For each prefix (k), each node has a link to another node Matching k bits Not matching on the k+1 bit  Each key from ID space mapped to a node with closest ID  Routing for a key: Bit correction Example: from 0101 for key 1001 0101  1110  1000  1000  1001  Lots of different algorithms with different properties Pastry, Tapestry, CAN, CHORD, SkipNet, etc. Load-balancing, robustness, etc.  A DHT can be viewed as a mesh of trees  Routes from all nodes to a particular key form a tree  Different trees for different keys

9. November 17, 2015Department of Computer Sciences, UT Austin9 DHT trees as aggregation trees  Key 111 010 001 100 101 110 111 011000 1xx 11x 111

10. November 17, 2015Department of Computer Sciences, UT Austin10 DHT trees as aggregation trees  Keys 111 and 000 010 001 100 101 110 111 011000 1xx 11x 111 010 001 100 101 110 111 011000 0xx 00x 000

11. November 17, 2015Department of Computer Sciences, UT Austin11 API – Design Goals  Expose scalable aggregation trees from DHT  Flexibility: Expose several aggregation mechanisms  Attributes with different read-to-write ratios “CPU load” changes often: a write-dominated attribute –Aggregate on every write  too much communication cost “NumCPUs” changes rarely: a read-dominated attribute –Aggregate on reads  unnecessary latency  Spatial and temporal heterogeneity Non-uniform and changing read-to-write rates across tree Example: a multicast session with changing membership  Support sparse attributes of same functionality efficiently  Examples: file location, multicast, etc.  Not all nodes are interested in all attributes

12. November 17, 2015Department of Computer Sciences, UT Austin12 Design of Flexible API  New abstraction: separate attribute type from attribute name  Attribute = (attribute type, attribute name)  Example: type=“fileLocation”, name=“fileFoo”  Install: an aggregation function for a type  Amortize installation cost across attributes of same type  Arguments up and down control aggregation on update  Update: the value of a particular attribute  Aggregation performed according to up and down  Aggregation along tree with key=hash(Attribute)  Probe: for an aggregated value at some level  If required, aggregation done to produce this result  Two modes: one-shot and continuous

13. November 17, 2015Department of Computer Sciences, UT Austin13 Scalability and Flexibility  Install time aggregation and propagation strategy  Applications can specify up and down  Examples Update-Local up=0, down=0 Update-All up=ALL, down=ALL Update-Up up=ALL, down=0  Spatial and temporal heterogeneity  Applications can exploit continuous mode in probe API To propagate aggregate values to only interested nodes With expiry time, to propagate for a fixed time  Also implies scalability with sparse attributes

14. November 17, 2015Department of Computer Sciences, UT Austin14 Scalability and Flexibility  Install time aggregation and propagation strategy  Applications can specify up and down  Examples Update-Local up=0, down=0 Update-All up=ALL, down=ALL Update-Up up=ALL, down=0  Spatial and temporal heterogeneity  Applications can exploit continuous mode in probe API To propagate aggregate values to only interested nodes With expiry time, to propagate for a fixed time  Also implies scalability with sparse attributes

15. November 17, 2015Department of Computer Sciences, UT Austin15 Scalability and Flexibility  Install time aggregation and propagation strategy  Applications can specify up and down  Examples Update-Local up=0, down=0 Update-All up=ALL, down=ALL Update-Up up=ALL, down=0  Spatial and temporal heterogeneity  Applications can exploit continuous mode in probe API To propagate aggregate values to only interested nodes With expiry time, to propagate for a fixed time  Also implies scalability with sparse attributes

16. November 17, 2015Department of Computer Sciences, UT Austin16 Scalability and Flexibility  Install time aggregation and propagation strategy  Applications can specify up and down  Examples Update-Local up=0, down=0 Update-All up=ALL, down=ALL Update-Up up=ALL, down=0  Spatial and temporal heterogeneity  Applications can exploit continuous mode in probe API To propagate aggregate values to only interested nodes With expiry time, to propagate for a fixed time  Also implies scalability with sparse attributes

17. November 17, 2015Department of Computer Sciences, UT Austin17 Scalability and Flexibility  Install time aggregation and propagation strategy  Applications can specify up and down  Examples Update-Local up=0, down=0 Update-All up=ALL, down=ALL Update-Up up=ALL, down=0  Spatial and temporal heterogeneity  Applications can exploit continuous mode in probe API To propagate aggregate values to only interested nodes With expiry time, to propagate for a finite time  Also implies scalability with sparse attributes

18. November 17, 2015Department of Computer Sciences, UT Austin18 Autonomy  Systems spanning multiple administrative domains  Allow a domain administrator control information flow Prevent external observer from observing the information in the domain Prevent external failures from affecting the operations in the domain  Support for efficient domain wise aggregation  DHT trees might not conform  Autonomous DHT  Two properties Path Locality Path Convergence  Reach domain root first 010 001 100 101 110 111 011 000 Domain 1 Domain 2 Domain 3

19. November 17, 2015Department of Computer Sciences, UT Austin19 Outline  SDIMS: a distributed operating system backbone  Aggregation abstraction  Our approach  Leverage Distributed Hash Tables  Separate attribute type from name  Flexible API  Prototype and evaluation  Conclusions

20. November 17, 2015Department of Computer Sciences, UT Austin20 Prototype and Evaluation  SDIMS prototype  Built on top of FreePastry [Druschel et al, Rice U.]  Two layers Bottom: Autonomous DHT Top: Aggregation Management Layer  Methodology  Simulation Scalability Flexibility  Prototype Micro-benchmarks on real networks –PlanetLab –CS Department

21. November 17, 2015Department of Computer Sciences, UT Austin21  Methodology  Sparse attributes: multicast sessions with small size membership  Node Stress = Amt. of incoming and outgoing info  Two points  Max Node Stress an order of magnitude less than Astrolabe  Max Node Stress decreases as the number of nodes increases Simulation Results - Scalability Astrolabe 256 Astrolabe 4096 Astrolabe 65536 SDIMS 256 SDIMS 4096 SDIMS 65536

22. November 17, 2015Department of Computer Sciences, UT Austin22 Simulation Results - Flexibility  Simulation with 4096 nodes  Attributes with different up and down strategies Update-Local Update-Up Update-All Up=5, down=0 Up=all, down=5

23. November 17, 2015Department of Computer Sciences, UT Austin23 Prototype Results  CS department: 180 machines (283 SDIMS nodes)  PlanetLab: 70 machines Department Network 0 100 200 300 400 500 600 700 800 Update - AllUpdate - UpUpdate - Local Latency (ms) Planet Lab 0 500 1000 1500 2000 2500 3000 3500 Update - AllUpdate - UpUpdate - Local

24. November 17, 2015Department of Computer Sciences, UT Austin24 Conclusions  SDIMS – basic building block for large-scale distributed services  Provides information collection and management  Our Approach  Scalability and Flexibility through Leveraging DHTs Separating attribute type from attribute name Providing flexible API: install, update and probe  Autonomy through Building Autonomous DHTs  Robustness through Default lazy reaggregation Optional on-demand reaggregation

25. November 17, 2015Department of Computer Sciences, UT Austin25 For more information: http://www.cs.utexas.edu/users/ypraveen/sdims