1. Intelligent Storage Project Keqiang Wu October 05, 2002

2. Intelligent Storage Project System Architecture –Boot-up –Arrival –Departure –Crash Concurrency Control –Group states –Speculation

3. ISD Backup Object migration wireless CA MN NY GL M MM M M M MM M M M MM M M Divide / Merge GL M M M M ISD : intelligent storage device GL : group leader M : group member

4. System Architecture Performance vs. Scalability –Internet: more on scalability Pastry (PAST): each node is randomly assigned a 128-bit node identifier. –Enterprise: performance >(?) scalability Directory service (?) Hierarchy base (?)

5. System Architecture: Boot-up Assumptions –IP addresses of ISD are well-known –Landmarks of ISD for geographical locations –M ISD, N nodes, L landmarks (M >=L) –Configuration parameter: K

6. System Architecture: Boot-up 1.Each node multicasts a REGISTRY msg to all landmarks. 2.Landmarks send back ACK. 3.A node multicasts LEADER-CLAIM msg (with RTT to the closest landmark) to all nodes as soon as it receives ACK. 4.Based on RTT, the closest node wins. 5.A node responds every leader with MEMBERSHIP-APPL (include RTT). 6.Based on RTT and the configuration parameter k, a leader sends APPROVEMENT/REJECTION msgs to others and builds up temporary Member List with IP addresses. 7.A node might receive >1 APPR msgs. Based on RTT between it and leaders (or landmarks), it sends ACCEPTION/REJECTION to the leaders.

7. System Architecture: Boot-up 8.A node might receive only REJECTION msg. It waits a certain amount of time and sends every leader with MEMBERSHIP- APPL (include RTT) again. If still fails, it will claim as a leader … 9.A leader finalize its Member List (table). 10.The leadership within each group is re-elected based on i.RTT from the node to the corresponding landmark. ii.Who is most powerful (CPU clock rate, memory space, etc.). iii.Willingness or fairness (?) 11.A Leader acquires information of local ISD, other local leaders, and remote leaders (not all, one per one remote landmark), (or the landmarks of remote ISD).

8. System Architecture: Arrival 1.A new node multicasts to all ISD landmarks and figures out which landmark is closest to itself. 2.The node multicasts MEMBERSHIP-APPL to all leaders around that closest ISD landmark (those leaders should have the same landmark as their local landmark). 3.The leaders respond with APPROVEMENT/REJECTION msgs. 4.The node decides which group to join based on RTT if it receives >1 APPROVEMENT msg. 5.If the node receives only REJECTION msg. It tries again later. If still fails, it will claim as a leader … 6.Exception: The leader who accepts the new node might be faraway. To ensure good locality, each leader periodically checks its members and gives those who are not geographically close the second chance in choosing a group.

9. System Architecture: Departure 1.A departure member only need to inform its leader. Then the leader updates its MEMBER LIST. 2.A departure leader lets the group choose a new leader (or there has already a deputy leader for fault tolerance), and passes its management information to the new leader. 3.The new leader informs the other leaders which store the information of the departure leader to update.

10. System Architecture: Crash 1.A member crashes: Each leader periodically checks with its members. If it does not receive response from a member for a certain amount of time, it assumes the member has failed and removes the member from the MEMBERSHIP LIST. All pending requests related to the crashed member will be cancelled (or continue for future recovery ?). 2.A leader crashes: Each member within a group has the information of the leader. If it has not received the checking information from the leader, it assumes the leader has failed and multicasts LEAD-CLAIM msg (includes information of the previous leader). Only those have the same previous leader respond. The new leader obtains all necessary information by contacting other leaders, ISD landmarks, and local ISD.

11. Other Issues: –Migration –Replication –Update of ISD System Architecture

12. Concurrency Control ISD-Based Directory Protocol –A client at CA requests an object. This request is passed to the leader. The leader responds with IP address of ISD if it finds the object in one of its ISD. Otherwise, the leader multicast the request to other leaders in CA; at the same time, it contacts one of leaders in MN and NY. CA ISD M L MN ISD M L NY ISD M L cache Hub memory MP o o o

13. Concurrency Control ISD-Based Directory Protocol –Each ISD records a state (unowned, shared, exclusive, 3 busy states, 1 migration state) table for each object it has. –A leader/member/ISD speculatively responds with data if it has a copy of the object which is requested by a client. At the same time, the leader/member/ISD interventions the home ISD of the requested object. –The client has two options: i.continue execution after receiving the speculative response but need to keep a log. ii.keep the speculative response but wait for the confirmation from the home ISD. –Upon the response from the home ISD, the client: i.removes the log if the speculation is correct, or undoes what has been done on the log and restarts based on the updated data. ii.continues execution if the speculation is correct, or throws the speculative object and continue execution with the updated object.

14. Concurrency Control Request Handling Examples MLHL’ 1 2 3b 3a A read/read-exclusive request to an object in unowned state at home ISD. MLHL’ 1 2 3b: intervention 3a: speculative reply A read/read-exclusive request to an object which a remote leader has a copy. 3c 4b: revision 4a: shared/exclusive response

15. Concurrency Control More Request Handling Examples –Read –Write –Write-back –Replacement Protocol Refinement