1. By Nitin Bahadur Gokul Nadathur Department of Computer Sciences University of Wisconsin-Madison Spring 2000

2. Multicast / Reduction Trees2 Talk Outline Motivation and Goals General Architecture of the middleware Components of the middleware Providing reliability - handling of node failures Applications developed using the middleware Performance Conclusions and possible extensions

3. Spring 2000Multicast / Reduction Trees3 Motivation and Goals A middleware for an application with Master - Worker paradigm Scalable framework for communication and computing client response (“Reduction”) Unicast does not scale - so use multicast Introducing reduction operations dynamically in clients A general framework for communication among clients

4. Spring 2000Multicast / Reduction Trees4 The Big Picture... Master App ARTL Client App ARTL Client App ARTL Client App ARTL Sends queries Reduces results Hands back results to application Execute responses to queries Forward queries downstream Reduces incoming results Sends reduced results to master Executes responses to queries Sends back results towards master

5. Spring 2000Multicast / Reduction Trees5 ART - Library Architecture Network ARTL Communication Layer Event Handler Application API Framework for processing messages Incoming Packet ARTL specific message Application specific callbacks Reduction functions ARTL messages :1. Query from master 2. Response from downstream nodes Outgoing message Application

6. Spring 2000Multicast / Reduction Trees6 ART - Library Architecture Network ARTL Communication Layer Event Handler Application API Framework for processing messages Incoming Packet ARTL specific message Application specific callbacks Reduction functions ARTL messages :1. Query from master 2. Response from downstream nodes Outgoing message Application

7. Spring 2000Multicast / Reduction Trees7 Communication Subsystem Connection Setup –Connect nodes as a Binomial tree Send and receive ARTL and application messages Detect node failure and act accordingly Integrate restarted node in current tree structure

8. Spring 2000Multicast / Reduction Trees8 Why use Binomial Tree Master App Client App 1 2 2 Master App Client App Binomial Tree Query Propagation time = 2 Unicast Mechanism Query Propagation time = 3 2 3 1

9. Spring 2000Multicast / Reduction Trees9 Reduction 15327684 Reduction at 5 and 3 Responses Example Reduction operations: Min(), Max()

10. Spring 2000Multicast / Reduction Trees10 15327684 Tree connection setup

11. Spring 2000Multicast / Reduction Trees11 15327684 Tree Setup - Phase I TCP connection setup

12. Spring 2000Multicast / Reduction Trees12 15327684 Tree Setup - Phase II TCP connection setup

13. Spring 2000Multicast / Reduction Trees13 15327684 Tree Setup - Phase III TCP connection setup

14. Spring 2000Multicast / Reduction Trees14 Inter node communication Unicast and multicast data transmission ARTL receives application messages for which no receive has been posted –these are sent to a callback function registered by application ARTL receives data on behalf of application when application explicitly posts a receive DataARTL Header

15. Spring 2000Multicast / Reduction Trees15 ART - Library Architecture Network ARTL Communication Layer Event Handler Application API Framework for processing messages Incoming Packet ARTL Encapsulated message Application specific callbacks Reduction functions ARTL messages :1. Query from master 2. Response from downstream nodes Outgoing message Application

16. Spring 2000Multicast / Reduction Trees16 Reduction Functions Implemented as Shared objects Sent to client during Setup phase Each reduction function is associated with a particular response it reduces

17. Spring 2000Multicast / Reduction Trees17 Event Handler Network Table containing Query id and Callback information for currently registered queries Responses for the shaded entry from down stream nodes Reduced response sent upstream Event Handler Application Response Callback Run Queue of reduction/response operations Thread Pool

18. Spring 2000Multicast / Reduction Trees18 Multithreaded Architecture No prior Knowledge about behavior of reduction function Exploit concurrency - multiple processor per node Static Pool of threads - Creation and destruction of threads is bad (Firefly RPC)

19. Spring 2000Multicast / Reduction Trees19 Crash Reconfiguration 15327684

20. Spring 2000Multicast / Reduction Trees20 Crash Reconfiguration 1537684 Crash Reconfiguration at depth 1

21. Spring 2000Multicast / Reduction Trees21 Crash Reconfiguration 1537684 Crash Reconfiguration at depth 2

22. Spring 2000Multicast / Reduction Trees22 15327684 Crash Reconfiguration Crash Reconfiguration at depth 1

23. Spring 2000Multicast / Reduction Trees23 1327684 Crash Reconfiguration Crash Reconfiguration at depth 1

24. Spring 2000Multicast / Reduction Trees24 Crash Detection Break in TCP connection with parent/child –a signal is received at the other end of connection Use of periodic refresh messages to inform parent that child is up and running –useful in WAN environments

25. Spring 2000Multicast / Reduction Trees25 Crash Handling Parent of node down informs master All nodes are informed of a node failure Master recomputes tree –If leaf node down, then no problem –If intermediate node down, some reconfiguration is required

26. Spring 2000Multicast / Reduction Trees26 Node Restart Restarted node contacts master to tell it about restart Master sends it current state of network and the shared object(s) All nodes are informed of a node restart Master recomputes tree and informs the new node’s parent about its new child Parent and child establish connections

27. Spring 2000Multicast / Reduction Trees27 SysMon - A System monitor Monitors the load average from /proc displays Min, Max and average loads Per-node load is also displayed ARTL Reduction operations : Min, Max and Average

28. Spring 2000Multicast / Reduction Trees28 SysMon - A System monitor Node failures are detected and SysMon pops up an alert

29. Spring 2000Multicast / Reduction Trees29 File Transfer Application Transfers a file from master to all clients File can be executed at clients (if required) –execution can be instantaneous on receiving file –execution can be delayed until all nodes have received the file

30. Spring 2000Multicast / Reduction Trees30 File Transfer Performance

31. Spring 2000Multicast / Reduction Trees31 Total Startup Time vs Number of Nodes Client processes started using ssh on different machines

32. Spring 2000Multicast / Reduction Trees32 Conclusions and Extensions A middleware for dynamic operations Support for crash detection, recovery and dynamic processes Demonstrated near optimal speedup using real applications Making response function dynamic - active services Differential scheduling in thread scheduler for QoS Making dynamic code secure