1. Presented at University of Alabama CIS, Birmingham Monday, April 9, 2001 Patterns-based Fault Tolerant CORBA Implementation for Predictable Performance Aniruddha Gokhale agokhale@lucent.com In collaboration with Balachandran Natarajan (bala@cs.wustl.edu)bala@cs.wustl.edu Douglas C. Schmidt (schmidt@uci.edu)schmidt@uci.edu Shalini Yajnik (shalini@precache.com)

2. Aniruddha GokhalePatterns for FT CORBA 2 Motivation Distributed applications are becoming more complex & mission-critical Increasing demand for cots- based multi-dimensional quality-of-service (QoS) support E.g., Simultaneous requirements for efficiency, predictability, scalability, security, & dependability Key open challenge is QoS- enabled dependability

3. Aniruddha GokhalePatterns for FT CORBA 3 Promising Solution: Fault Tolerant (FT) Distributed Object Computing Middleware Challenges Limitations of non-OO FT strategies that focus on application processes Techniques based on process- based failure detection & recovery are not applicable to distributed object computing applications due to: 1.Overly coarse granularity 2.Inability to restore complex object relationships 3.Restrictions on process checkpointing & recovery

4. Aniruddha GokhalePatterns for FT CORBA 4 Overview of Fault Tolerant CORBA Overview Provides a standard set of CORBA interfaces, policies, & services Entity Redundancy of objects is used for fault tolerance via Replication Fault detection & Recovery from failure Features Inter-Operable Group References (IOGR) Replication Manager Fault Detector & Notifier Message Logging for recovery Fault tolerance Domains

5. Aniruddha GokhalePatterns for FT CORBA 5 Interoperable Object Group References Composite & enhanced Interoperable Object Reference (IOR) for referencing server object groups Comprises one or more TAG_INTERNET_IOP profiles, which in turn must contain a TAG_FT_GROUP and zero or more TAG_IIOP_ALTERNATE_ADDRESS components TAG_PRIMARY component in at most one TAG_INTERNET_IOP profile Client ORBS operate on IOGRs in the same way as with IORs

6. Aniruddha GokhalePatterns for FT CORBA 6 DOORS & FT-CORBA DOORS is a “Distributed OO Reliable Service” developed prior to FT-CORBA Uses the service strategy to provide FT to CORBA objects Patterns and mechanisms in DOORS were integrated into FT-CORBA standard DOORS implements most of FT-CORBA standard Focus on passive-replication Available as open-source for non-commercial use from Lucent Runs atop the TAO open-source real-time ORB www.theaceorb.com

7. Aniruddha GokhalePatterns for FT CORBA 7 8.Client sends requests to the primary 2. RM delegates replica creation to local factories 4.The local factories send the replica IOR’s to the RM for it to create the IOGR 5.The RM registers the IOGR with a CORBA Naming Service (NS) 7.Clients contact the NS for IOGR FT-CORBA Component Interaction 1.External object asks RM to set properties for replica group and create it 6.The RM asks fault detectors to initiate fault monitoring of replicas 3.The local factories create CORBA objects

8. Aniruddha GokhalePatterns for FT CORBA 8 2.Detector propagates fault to Notifier 4.RM promotes backup to primary 5.RM requests local factory to create a new backup and gets new IOR 7.RM registers new IOGR with NS Fault Detection and Recovery 1.Fault detector detects failure of primary 6.RM creates new IOGR and informs all replicas of it 3.Notifier pushes fault to RM PRIMARYBACKUP 8.Client sends request to old primary 9.Old primary throws exception LOCATION_FORWARD 10.Client sends request to new primary PRIMARYBACKUP

9. Aniruddha GokhalePatterns for FT CORBA 9 ORB Core Optimizations Optimization Opportunities to Improve Fault Tolerant CORBA Performance CORBA Service Optimizations Efficient IOGR parsing & connection establishment Reliable handling & ordering of GIOP messages Predictable behavior during transparent connection establishment & retransmission Tracking requests with respect to the server object group Support for dynamic system configuration Bounded recovery time Minimize overhead of FT CORBA components

10. Aniruddha GokhalePatterns for FT CORBA 10 Analysis Failure detection time increases with the polling interval Average failure detection time is half the polling interval Challenge Choosing small polling interval Minimize message overhead Effect of Polling Interval on Failure Detection Times Fault detection time measured as the time between the failure of replica & the FaultDetector detecting failure

11. Aniruddha GokhalePatterns for FT CORBA 11 Effect of Polling Interval on Recovery Time Analysis Average failure detection time is half the polling interval Replica Group Management time is constant Challenge Minimize replica group management time Recovery Time = Failure detection time + Replica Group Management Time

12. Aniruddha GokhalePatterns for FT CORBA 12 Design patterns capture the static & dynamic roles & relationships in solutions that occur repeatedly Architectural patterns express a fundamental structural organization for software systems that provide a set of predefined subsystems, specify their relationships, & include the rules and guidelines for organizing the relationships between them Optimization principle patterns document rules for avoiding common design & implementation mistakes that degrade performance Patterns codify expert knowledge to help generate software architectures by capturing recurring structures & dynamics and resolving common design forces Overview of Patterns www.posa.uci.edu/

13. Aniruddha GokhalePatterns for FT CORBA 13 Decoupling Polling and Recovery Fault Detector Polling Thread Replica Fault Notifier Context Periodic polling & recovery request done in the same polling thread can block the thread Forces Must guarantee polling of other objects while recovery request is sent Must minimize concurrency overhead Solution Apply the Leader-Followers or AMI architectural pattern Solution Apply the Leader-Followers or AMI architectural pattern Problem Blocking can cause missed polls HANGS

14. Aniruddha GokhalePatterns for FT CORBA 14 Decoupling Recovery Initiation From Recovery Execution Context Replication Manager serializes failure reports Forces Bounded amount of time for failure recovery irrespective of number of failure reports Solution Apply the Active Object design pattern Solution Apply the Active Object design pattern Problem Reduced responsiveness

15. Aniruddha GokhalePatterns for FT CORBA 15 Supporting Interchangeable Behavior Solution Apply the Strategy design pattern Solution Apply the Strategy design pattern Context FT properties can be set statically (as defaults) or set dynamically Forces Need highly extensible services that can be composed transparently from configurable properties Problem Hard-coding properties make the FT- CORBA design inflexible & non- extensible

16. Aniruddha GokhalePatterns for FT CORBA 16 Consolidating Strategies Context. FT CORBA implementations can have many properties. e.g.,membership, replication, consistency, monitoring, # of replicas, etc. Forces Ensure semantically compatible properties Simplify management of properties Solution Apply the Abstract Factory design pattern Solution Apply the Abstract Factory design pattern Problem Risk of combining semantically incompatible properties

17. Aniruddha GokhalePatterns for FT CORBA 17 Dynamic Configuration Context There are many potential FT properties that can be used Forces The behavior of FT-CORBA properties should be decoupled from the time when they are actually configured Solution Apply the Component Configurator design pattern Solution Apply the Component Configurator design pattern Problem Static configuration of properties is inflexible & overly resource intensive

18. Aniruddha GokhalePatterns for FT CORBA 18 Context FT-CORBA mandates a hierarchical lookup of properties based on strings Property lookup is required during object group creation & recovery Forces Efficient lookups of properties guided by the order specified in the FT-CORBA standard Solution Use the Chain of Responsibility design pattern & Perfect Hashing optimizations Solution Use the Chain of Responsibility design pattern & Perfect Hashing optimizations Efficient Property Name-Value Lookup Problem Inefficient property lookup degrades QoS

19. Aniruddha GokhalePatterns for FT CORBA 19 Research Directions  Middleware for Ad hoc/Wireless networks  FT CORBA enhancements for JINI-like systems  CORBA Pluggable Protocol for Bluetooth devices  Middleware enhancements for 3G wireless/mobile internet  Fault tolerance  Sequenced Initialization and Recovery (dealing with object dependencies)  Handling failure groups and collocated groups  Fault Escalation strategies and Fault Analysis  Growth/degrowth, runtime upgrades  QoS-enabled framework of middleware components  Higher level middleware framework shielding applications from lower level middleware  Multi-dimensional QoS support  Patterns-based architecture of plug & play components  Code generation tools for repetitive tasks

20. Aniruddha GokhalePatterns for FT CORBA 20 Concluding Remarks Researchers & developers of distributed systems face common challenges, e.g.: The application of patterns, frameworks, & components can help to resolve these challenges Carefully applying these techniques can yield efficient, scalable, predictable, dependable, & flexible middleware & applications Connection management, service initialization, error handling, flow control, event demuxing, distribution, concurrency control, fault tolerance, synchronization, scheduling, & persistence

21. Aniruddha GokhalePatterns for FT CORBA 21 EXTRA SLIDES

22. Aniruddha GokhalePatterns for FT CORBA 22 Fault Tolerant Middleware Strategies Integration Strategy Modify middleware to support FT, e.g.: Orbix+Isis www.iona.com Electra www.softwired.ch/people/maffeis/ electra.html Obtrusive Interception Strategy Intercepts messages outside the ORB, e.g.: Eternal www.eternal-systems.com AQuA www.crhc.uiuc.edu/PERFORM/ AQuA.html Unobtrusive, but complex

23. Aniruddha GokhalePatterns for FT CORBA 23 Service Strategy FT as a higher-layer service, e.g., DOORS www.bell-labs.com/ org/11356/html/doors.html Unobtrusive, but requires standard ORB support Fault Tolerant Middleware Strategies (cont’d) Summary of FT Strategies Integration strategy requires extensive non- portable & non-standard modifications to an ORB Interception strategy provides out-of-band solution that can be inefficient (due to duplication of effort) and is hard to port across operating systems Service strategy requires some ORB modifications, but is now standardized…

24. Aniruddha GokhalePatterns for FT CORBA 24 Replication Manager Components Property manager Allows properties to be set for an object group Properties include replication style, membership style, consistency style, monitoring style, & number of replicas Generic factory Creates object groups & each member of the object group Used when the membership- style is infrastructure-controlled Object group manager Used by applications to create, add or delete members of an object group

25. Aniruddha GokhalePatterns for FT CORBA 25 Fault Detection & Notification Components Fault Detectors Detect faults using a pull- based or a push-based mechanism Fault Notifier Notified of any faults by the Fault Detector A Fault Notifier notifies its Replication Manager when it detects faults

26. Aniruddha GokhalePatterns for FT CORBA 26 Logging & Recovery Used in a infrastructure- controlled consistency style The mechanism intercepts & logs GIOP messages On failure detection, the messages can be “played back” for recovery Mechanism is transparent to the client application

27. Aniruddha GokhalePatterns for FT CORBA 27 FT CORBA Spec Requirements Preserving the CORBA Object Model with enhancements No single point of failure Transparent failover Transparent client redirection & reinvocation FT CORBA cannot handle commision faults or correlated faults

28. Aniruddha GokhalePatterns for FT CORBA 28 Caching Object References Context Replication Manager queries Naming Service for Fault Detector object reference Forces Minimize time spent in querying for object reference Solution Optimize for common case, store redundant info, eliminate gratuitous waste Solution Optimize for common case, store redundant info, eliminate gratuitous waste Problem Adds overhead of querying