1. Z. Kalbarczyk K. Whisnant, Q. Liu, R.K. Iyer Center for Reliable and High-Performance Computing Coordinated Science Laboratory University of Illinois at Urbana-Champaign 1308 W. Main St.; Urbana, IL 61801 ARMOR-based Hierarchical Fault/Error Management

2. University of Illinois at Urbana-Champaign Networked/Distributed Systems Key Questions n How do we integrate components with varying fault tolerance (detection and recovery) characteristics into a coherent high availability networked system? n How do you guarantee reliable communications? n How do you synchronize actions of dispersed processors and processes? n How do you contain errors (or achieve fail-silent behavior of components) to prevent error propagation? n How do you reconfigure the system in response to failures?

3. University of Illinois at Urbana-Champaign Failure Categories n Necessity to cope with machine (node), process, and network failures n A component specification defines what output should be produced in response to any sequence of inputs as well as the real-time interval within which this output should occur

4. University of Illinois at Urbana-Champaign Failure Categories (cont.) n Crash failures are a proper subclass of omission failures –a crash failure occurs when after a first omission to send/receive a message a process systematically omits to send/receive messages n Omission failures are a proper subclass of timing failures –a process which suffers an omission failure can be understood as having an infinite response time n Timing failures are a proper subclass of incorrect computation failures –a timing failure occurs when a process takes some action too soon or too late n Incorrect computation failures are a proper subclass of the class of all possible failures, the Byzantine or malicious failures –a faulty process may send spurious messages to other processes, may lie, may not respond to received messages correctly

5. University of Illinois at Urbana-Champaign What Do We Propose in Approaching the Problems? n ARMOR-based programming environment that provides –A process architecture offers flexibility in assigning functionality to specific processes, including –error detection and recovery techniques, that can be reconfigured according to dependability and application requirements scales according to the number of nodes available and to the number of applications simultaneously executing in the system. –A runtime environment provides external process management to applications allows fine-tuning of fault tolerance services provided to and embedded in the application. –Hierarchy of error detection and recovery to avoid single point(s) of failure to provide protection not only to the applications, but to the entities supporting detection and recovery services

6. University of Illinois at Urbana-Champaign What are ARMORs? n Adaptive Reconfigurable Mobile Objects of Reliability: –Multithreaded processes composed of replaceable building blocks. –Provide error detection and recovery services to user applications via three levels of interaction. n Hierarchy of ARMOR processes form runtime environment: –System management, error detection, and error recovery services distributed across ARMOR processes. –ARMOR runtime environment is self-checking. n ARMOR support for the application: –Completely transparent and external support. –Transparent extension of standard libraries. –Instrumentation with ARMOR API.

7. University of Illinois at Urbana-Champaign ARMOR Configuration Progress Indicator element HB element Checksum Element ARMOR Microkernel ARMOR Repository of Elements Text-segment signature element Range-check element Assertion check element Control flow signature element Data dependency checking element Data dependency checking element Text-segment signature element Checksum Element HB element Checkpoint element Progress Indicator element Checkpoint element

8. University of Illinois at Urbana-Champaign ARMOR Computation Model n Elements invoked through events called operations. n A thread consists of a sequence of operations that execute. n In response to an operation, element can: –Read/write thread variables that serve as input/output for operation. –Read/write element state. –Generate additional operations to be processed within thread. n Element-based detection and recovery: –Monitor generates operation when it detects an error. –Policy elements subscribe to notification operation, and generate sequence of operations to effect recovery. –Service elements carry out individual recovery steps. n Response to errors can be reconfigured by changing policy elements in ARMORs. element opAction1 opAction3 opAction2

9. University of Illinois at Urbana-Champaign Node ARMOR Runtime Environment n Various kinds of ARMORs execute in environment depending upon requirements. n Distribution of detection and recovery responsibilities makes environment resilient to ARMOR failures. n Solutions scalable to one-node configuration. Daemon ARMOR ARMOR Manager ARMOR Daemon ARMOR Exec. ARMOR App. Daemon ARMOR Manager ARMOR Heartbeat ARMOR App. network Node Primary ARMOR Backup ARMOR App. multi-node solution single-node solution

10. University of Illinois at Urbana-Champaign Daemon Daemons n Each node in runtime environment executes a daemon. n Provide services to local ARMORs: –Install ARMORs on local node. –Detect ARMOR process crash/hang failures. –Channel for ARMOR-to-ARMOR communications. TCP Connection Mgmt. Named Pipe Mgmt. Process Mgmt. Detection Policy ARMOR Microkernel Process Mgmt. ARMOR Network Daemon Local ARMORs Remote daemons Node 1Node 2 Node 3

11. University of Illinois at Urbana-Champaign Managers n Manage a group of ARMORs. n Responsible for recovering failed ARMORs. n Contain information about each ARMOR: –Location in the network. –Current configuration. –Recovery policy. –Associated application. n Detect and recover from node failures. n Allocate nodes (including spares) for application and for ARMOR processes. n Interface with user. n Manager functionality can be consolidated into one Manager ARMOR or distributed across hierarchy of Manager ARMORs.

12. University of Illinois at Urbana-Champaign Hierarchy of Error Detection & Recovery – Attributes (1) n Adaptivity and composability of individual levels. –Detection and recovery composition and invocation at the individual levels should be customizable to meet. the application’s needs, the types of faults being experienced in the system, the reliability characteristic desired –Applications with varying availability requirements should coexist in the same environment –Detection levels should allow to be: selectively turned on or off independent so that they can be composed in various ways

13. University of Illinois at Urbana-Champaign Hierarchy of Error Detection & Recovery – Attributes (2) n Intra-level interactions –interactions between techniques placed within each level should be evaluated taking into account: cost, coverage, and intrusiveness factors e.g., placing assertion checks in certain points of the application code, may not required to generate control flow signatures for that portion of the code. n Inter-level interactions –interactions between error detection and recovery levels should be carefully defined to eliminate redundant invocation of multiple detection mechanisms. –errors that escape a given level should be detectable by higher levels n Recovery responsibilities –an appropriate recovery strategy should be selected based upon the failure and circumstances of the failure event –avoid a competition during error recovery – make sure that one and only one entity is responsible for recovery of a failed process or node

14. University of Illinois at Urbana-Champaign Hierarchy of Error Detection & Recovery Increasing overhead Detection Watchdog timer (livelock detection) Built-in assertion checks Control and data flow check Recovery Restart a process/thread Hardware reset ARMOR Error Detection & Recovery Process Inside ARMOR process Layer 2: Layer 3: Detection Progress indicator Smart heartbeats Data audits; OS detection Recovery Checkpointing/Rollback Process restart on the same node Layer 1: Detection Signature exchange between processes for consistency check Global heartbeats Recovery Checkpointing/Rollback Process migration/restart Masking n Techniques encapsulated in separate elements n Can be selectively turned on or off, inserted or removed n Arranged in a hierarchy of layers Node At the node Network Between ARMORs

15. University of Illinois at Urbana-Champaign ARMOR Applications  Base station controller: protecting call-processing application and database of digital mobile telephone network controller.  Embedded wireless applications:  protecting wireless communication channel through ARMOR- based proxies.  Providing automated detection and recovery to wireless telephones and servers  Network services: DHCP (Dynamic Host Configuration Protocol).  Spaceborne applications: runtime environment for protecting distributed spaceborne applications.

16. ARMOR-Based Fault Management in RTES Environment – Design Options

17. University of Illinois at Urbana-Champaign Manager on DSP n Local Managers: –Execute on dedicated DSP per board. –Detect and recover from errors localized to board. Mgr App MgrApp Level 1 DSP Farm Level 2/3 Linux Farm Daemon Exec ARMOR App Node Board App Daemon Exec ARMOR Node Daemon Region Mgr. Node Board n Regional Managers: –Execute on Linux clusters. –Handle recovery that spans multiple boards. com

18. University of Illinois at Urbana-Champaign Manager on PC n Local Managers execute on PC assigned to board or group of boards. Mgr App Daemon Exec ARMOR App Node Board App Daemon Exec ARMOR Node Daemon Region Mgr. Node BoardLinux/Win32 App Mgr Linux/Win32 com

19. University of Illinois at Urbana-Champaign App Daemon Exec ARMOR Node ARMOR-based Manager – Design Details (1) n Local Managers are ARMOR processes: –Reconfigurable monitoring functionality, detection policy, recovery policy. –Communicate with Linux farm through common ARMOR infrastructure. App Board Daemon Region Mgr. Node AppARMOR com Linux/Win32 Daemon ARMOR Microkernel Recovery Policy Local Manager ARMOR App ARMOR Interface DSP Interface Daemon

20. University of Illinois at Urbana-Champaign ARMOR-based Manager – Design Details (2) n All functionality found in replaceable elements. n Individual ARMORs can be customized based upon role they play in the system: –Local Manager ARMORs include element to interface with DSP. –Daemon ARMORs contain elements to communicate with local ARMORs. –Execution ARMORs contain elements to oversee user application. n All ARMORs consists of “microkernel” used to add elements, remove elements, communicate among elements. n Each element found in separate shared library: –Elements are explicitly loaded by microkernel through dl_open() and dl_sym(). –Dynamic reconfiguration can be done on demand. n Elements subscribe to event messages that they are designed to process. n Tcl interface used to construct messages that are sent to ARMORs.