Presentation is loading. Please wait.

Presentation is loading. Please wait.

Self-Stabilizing Systems as a Base for Autonomic Computing Shlomi Dolev Yinnon Haviv, Reuven Yagel, Olga Brukman.

Similar presentations


Presentation on theme: "Self-Stabilizing Systems as a Base for Autonomic Computing Shlomi Dolev Yinnon Haviv, Reuven Yagel, Olga Brukman."— Presentation transcript:

1 Self-Stabilizing Systems as a Base for Autonomic Computing Shlomi Dolev Yinnon Haviv, Reuven Yagel, Olga Brukman

2 Trustworthy Systems: Why Is It So Hard? Corbató’91: "It almost goes without saying that ambitious systems never quite work as expected“ "You must pay extreme attention to detail here. One wrong bit will make things fail… " From Pentium’s manual: “… if the ESP or SP register is 1 when the PUSH instruction is executed, the processor shuts down due to a lack of stack space. No exception is generated to indicate this condition"

3 Mars Rover - Spirit …The Spirit rover has a radiation-hardened R6000 CPU from Lockheed-Martin Federal Systems…The operating system is Wind River Systems' Vx-Works.. …attempted to allocate more files than the RAM-based directory structure could accommodate. That caused an exception, which caused the task that had attempted the allocation to be suspended… …Spirit fell silent, alone on the emptiness of Mars, trying and trying to reboot

4 Linux and Windows do not Stabilize

5 Self-Stabilization Self-healing, Self-managing, Self-* Recovery Oriented Computing [Berkeley, Stanford] Autonomic Computing [IBM] Self-Stabilization Self-Stabilizing algorithm for mutual exclusion in a ring topology [Dijkstra’74]

6 Well Established Theory !

7 Self-Stabilizing Systems Elegant fault tolerant approach. Started at any state, the system convergences to a desired behavior. Generally used in distributed systems. Routing, clock synchronization, leader election, etc. Overcome transient faults in the system. Transient faults: soft-errors (“98% of RAM errors are soft errors”), wrong CRC during communication etc.

8 Self-Stabilization The combination and type of faults cannot be totally anticipated in on-going systems Any on-going system must be self stabilizing (or manually monitored) E L

9 First Self-Stabilizing Algorithm: Token Passing

10 Token Passing 1 P 1 : do forever 2 if x 1 =x n then 3 x 1 :=(x 1 +1)mod(n+1) 4 P i (i ≠ 1): do forever 5 if x i ≠x i- 1 then 6 x i :=x i-1

11 Token Passing Cont. Surely works when we start in x 1 = x 2 = … = x n = 0. One processor may change a state at a time. {0; 0; 0; 0; 0}; {1; 0; 0; 0; 0}; {1; 1; 0; 0; 0}; {1; 1; 1; 0; 0}; {1; 1; 1; 1; 0}; {1; 1; 1; 1; 1}; {2; 1; 1; 1; 1}; {2; 2; 1; 1; 1}; {2; 2; 2; 1; 1}; {2; 2; 2; 2; 1}; {2; 2; 2; 2; 2} …

12 Token Passing: Faults Transient fault, soft errors, wrong CRC, unexpected temporal severe conditions, etc. Assigns each processor with an arbitrary state (in the range of its state space). For example {3; 4; 4; 1; 0}. p 2 ; p 4 ; and p 5 have tokens! Will the system ever recover?

13 Token Passing: Automatic Recovery p 1 changes state infinitely often, Otherwise, let s 1 be the fixed state of p 1, p 2 eventually copies s 1 from p 1, then p 3 eventually copies s 1 from p 2, then... p n eventually copies s 1 from p n-1, then p 1 changes state. p 1 changes state in the order 4; 5; 0; 1; 2; 3; 4; 5; 0;...

14 Token Passing: Automatic Recovery Cont. In any initial state at least one state is missing, {4; 4; 1; 0; 2}, 3 and 5 are missing. Once p 1 reaches the missing state e.g., 5, all the processors must copy 5, before p 1 reads 5 from p n and changes state to 0.

15 Will It Stabilize With mod (n - 2)? Mod 3 {0,0,2,1,0} p 1 {1,0,2,1,0} p 5 {1,0,2,1,1} p 4 {1,0,2,2,1} p 3 {1,0,0,2,1} p 2 {1,1,0,2,1} +1 mod 3 !

16 Is Self-Stabilization a Toy?

17 Talk Outline Self Stabilizing Microprocessor [DH04] Self Stabilizing Operating System [DY04] Self-Stabilization Preserving Compiler[DH05] Self-Stabilizing Automatic Recoverer For Eventual Byzantine Software [BDK03] Recovery Oriented Programming[BD05]

18 Self-Stabilizing Microprocessor Overcoming Soft-Errors Shlomi Dolev and Yinnon A. Haviv 17th International Conference on Architecture of Computing Systems (ARCS)

19 Motivation Soft-Errors: Single Event Upsets (SEU) Caused by cosmic ray / other disruptions. Cause a logical gate to flip its content. Currently handled only in memories. Significant impact on the microprocessors.

20 Soft-Errors - Current Solutions Obtaining masking using probabilistic approaches: Information redundancy (ECC / Parity) Space redundancy Time redundancy Failure detection / recovery. Known solutions: IBM S-390 Compaq NonStop Himalaya IROC

21 Self-Stabilizing Microprocessor  Self-stabilizing algorithms assume that the microprocessor executes them. Soft-errors may cause the microprocessor to be stuck in a faulty state.  A microprocessor is self-stabilizes if: Started in any internal state, converges in a finite number of steps into the set of safe states. Microprocessor’s safe state – in which it performs “fetch-decode-execute” cycle

22 Proving Convergence  Proving that there exists no “bad” cycle in the transition graph of the microprocessor.  Too large ! (we must explore the entire graph)  Using an abstraction:~ Group together states in which the micro-code program counter is the same. a b c d e f k l i j h g D E F A B C

23 Self-Stabilizing Microprocessor: Summary Soft-errors are here to stay, we should: Design our systems to mask them. Self-stabilize following a non-masked error. We provide methodology for validating self- stabilization property of microprocessors.

24 Talk Outline Self Stabilizing Microprocessor [DH04] Self Stabilizing Operating System [DY04] Self-Stabilization Preserving Compiler[DH05] Self-Stabilizing Automatic Recoverer For Eventual Byzantine Software [BDK03] Recovery Oriented Programming[BD05]

25 Toward Self-Stabilizing Operating System (SOS) Shlomi Dolev and Reuven Yagel, SAACS’04 Workshop, Zaragoza

26 Talk Outline The first self-stabilizing algorithm (of Dijkstra) Self Stabilizing Microprocessor [DH04] Self Stabilizing Operating System [DY04] Self-Stabilization Preserving Compiler[DH05] Self-Stabilizing Automatic Recoverer For Eventual Byzantine Software [BDK03] Recover Oriented Programming[BD05]

27 Basic Directions Black-box Take existing OS (Unix, Windows, RTOS) Add stabilization layer Carefully tailoring a tiny kernel  Processor scheduling  Memory management  Device allocation

28 Assumptions Every configuration (processor/memory) is possible At least some program code is hardwired (in ROM) and is correct – Harvard Model Processor: Instruction manual (e.g. x86\IA-32) defines a transition function. Self-stabilizing [DH04]

29 Black Box Requirements: Defining a legal execution is usually impractical At least - restore original state (variables + code), infinitely often Periodic Reset Re-install and Execute Watchdog timer (self-stabilizing) Periodic processor reset During bootstraps OS reinstall from ROM Weak self-stabilization E = (c i, a i, c i+1, …., RRE, c 1, a 1, c 2, a 2, …., c i, a i, c i+1, …., RRE, c 1, a 1, c 2, a 2, …. Is it always acceptable? Alternative: Periodic re-install code only, add consistency check and enforcement

30 Tailored Kernel Tiny Scheduler Tiny Memory Manager Requirements: Self-stabilizing Fair Process stabilization preserving (e.g. validity of P.C. value)

31 Tiny SOS Scheduler ; increase task 10 mov word ax, [currentProc] 11 and ax, PROC_MASK... ; load task state... ;restore ip 52 mov ax, [bx+4] ;validate ip 53 and ax, IP_MASK 54 mov word [ss:STACK TOP], ax ;restore general registers 55 mov cx, word [bx+12] 56 mov dx, word [bx+14] 57 mov si, word [bx+16] 58 mov di, word [bx+18] ~70 lines of a real machine assembly code 16bit Real mode & 32bit Protected mode. Standard build and emulation tools (Nasm, ld, Bochs) Detailed proof of requirement preservation

32

33 Tiny SOS Memory Manager Requirements: Consistency of memory hierarchy Self-stabilization preservation

34 Any State Process(ing) Next Process Validated & Ready Clock tick / execute next Some Error Establish Scheduler Consistency Tiny SOS Scheduler

35 Any State Process(ing) Next Process Validated & Ready Clock tick / execute next NMI / load PC with scheduler handler Establish Scheduler Consistency Tiny SOS Scheduler

36 Sketch of Proof In every execution E, the code of the scheduler is started to be executed and is executed from the first instruction to the last instruction infinitely often In every execution E of the scheduler each process is executed infinitely often The self-stabilizing scheduler preservers stabilization of processes.

37 Talk Outline Self Stabilizing Microprocessor [DH04] Self Stabilizing Operating System [DY04] Self-Stabilization Preserving Compiler[DH05] Self-Stabilizing Automatic Recoverer For Eventual Byzantine Software [BDK03] Recover Oriented Programming[BD05]

38 Self-Stabilization Preserving Compiler Shlomi Dolev, Yinnon A. Haviv, Department of Computer Science Ben-Gurion University, Israel Mooly Sagiv, Department of Computer Science Tel Aviv University, Israel

39 Motivation Transient malfunctions. Single processor: Hardware glitches. Soft-Errors. Distributed environment: Processor crashes / recoveries. Link errors. Resulting in an unpredictable system state.

40 Coping with Transient Errors Masking (safety factor) achieved by: Information redundancy (e.g., ECC). Time/Space redundancy. (e.g., TMR) Self-Stabilization [Dijkstra74]: Assuming any system state (caused by errors). Recovering by converging into legal behavior. Existing algorithms for distributed tasks:  Routing, leader election, mutual exclusion, etc.

41 Self-Stabilizing Algorithms – a Solution to Soft-Errors? Self-Stabilizing algorithm assumes that the microprocessor executes it. Soft-Errors may cause the microprocessor to be stuck in a faulty state. Composition of self-stabilizing algorithms creates a self-stabilizing system. Make the microprocessor eventually fetch-decode- execute machine code.

42 The Gap. Need a transformation between: Input program P written in a high abstraction language, e.g., (D)ASM. Output program Q in a machine language, say, JVM. Existing compilers? P and Q behaves the same when started in the initial state. What if Q reaches an unexpected state due to soft-error experienced by microprocessor?

43 Trivial Example A statement of the form: For each i in {0..9} do f(i) May be compiled to  Start with cx=12 inside the loop… Moreover: Any runtime mechanism can get stuck / inconsistent. mov ax, 10 mov cx, 0 loop1: push cx call f inc cx cmp cx,ax jne loop

44 Stabilization Preserving Compiler – a closer look  State space of P Ensuring that Q eventually behaves as P:  State space of Q

45 Self-Stabilization Preserving Compiler: Summary Front end of compiler established.  Typed version of ASM.  JavaCC as a parser generator. Interpreter (used as a model). Fast stabilization vs. optimizations. Self Stabilization preserving compiler. Language with clear semantics from any state. Innovative demands from compiler.

46 Talk Outline Self Stabilizing Microprocessor [DH04] Self Stabilizing Operating System [DY04] Self-Stabilization Preserving Compiler[DH05] Self-Stabilizing Automatic Recoverer For Eventual Byzantine Software [BDK03] Recover Oriented Programming[BD05]

47 Self-Stabilization and Evolving Systems Real world systems cannot be verified exhaustively… We enforce safety and live-ness specifications Contract between the client, project manager and programmers, that is checked on line! Make sure that the additional (thin) monitoring and recovering layer is self-stabilizing A change can be made to the implementation/specification to support evolving environments

48 Self-Stabilizing Recoverer for Eventual Byzantine Software Olga Brukman, Shlomi Dolev Department of Computer Science Ben-Gurion University, Israel Hillel Kolodner, Haifa Research Labs IBM, Israel

49 Software Contains Bugs Heisenbugs, corrupt states, leaked resources are common… Correct and faultless SW is hard Long-lived running programs, e.g., OS Usually software is tested when starting from initial state and considering limited time scenarios.

50 Fault Model Reflecting Reality Software packages can be trusted to work as required after restart. Eventual Byzantine software. System administrators and users use reboot to deal with faults.

51 Middleware Architecture OS Kernel OMR 1 2 … n

52 Monitor-Restarter for Process and Subsystem 1 2 …

53 Restart Actions – Mature Approach Subsystem waits for completion of a restart of its components. Restart action may vary, depending on component internal state. Reschedule Roll-back Kill & Restart Few restart attempts with more drastic restart actions.

54 Computational Model: rsf-execution An execution E is rsf (restart supporting fair)- execution iff E is a fair execution in which every subsystem sub i that is initialised during E respects its specification function ss i. Requirement: Every rsf-execution E has a suffix in which the system respects its specification function ss.

55 Tools for Implementation – Black Box Approach Software package is a black box. Package is monitored by recording it’s IO (e.g., strace in Linux). Monitors are independent of specific implementation

56 Tools for Implementation – Transparent Box Approach Software package implementation tool is known. Run-Time Reflection tools are used to monitor and restart the package. Possible in Java, C++, CORBA, COM.

57 Practical Experience: Printers Problem Corrupted pdf, doc or ps file sent to printing server. Printer can’t print the file. Cause retries by printing server Printer is “stuck” on one job. Predicate for printing server:  Restrict number of retries, try format conversions, send error message to user.

58 Self-Stabilizing Automatic Recoverer: Summary Theory foundations of self-stabilization and restart techniques could serve as a basis for the new paradigms. General framework for design and correctness proof for autonomic recoverer. Printers experience coordinated with IBM.

59 Recovery Oriented Programming Olga Brukman and Shlomi Dolev Department of Computer Science Ben-Gurion University, Israel

60 Towards Robust Software Programming Structural programming, OOD, Design Patterns… Testing and debugging Unit testing [JUnit, CppUnit]… Design By Contract (Eiffel) … Formal specification languages ASM, IO Automata, NURPL Model checking Online recovery ROC [PBB02]. Self-Stabilizing Autonomic Recoverer for Eventual Byzantine Software [BDK03] - black box software packages.

61 Our Contribution Program invariants derived from design specifications. Checked every time invariant variables are updated. Automatic code generation for invariant verification and recovery upon invariant violation. Invariants are verified during runtime. Change of invariant variable is pre-checked in sand- box. Violation is prevented and replaced with a recovery action.

62 Our Contribution Cont. Recovery action is chosen depending on the current state and history.  Roll back & resume.  Wait.  Reschedule.  Kill & restart.

63 External Monitoring Monitoring the whole task to avoid transient faults occurrence after which invariant variables are not changed ( and no invariant checks are done) liveness problem – monitor over time

64 Producer Consumer Example Producer, Consumer – threads Queue – a circular bounded length queue int queue[size] int start – position of current first element in the queue int end – position of the first empty place in the queue boolean empty ProducerConsumer Queue

65 Invariants for Queue start, end are in some values range ({0,..,size-1}) Queue is not empty iff start != end A possible invariant is: (start in {0,..,size-1}, end in{0,.., size-1}) && (start != end => empty == false)

66 Given Implementation Code class Queue { private int[] queue; private int start, end, size; private boolean empty; public Queue(int size){ queue = new int[size]; start = 0; end =0; empty= true; } public synchronized int dequeue() { int result; while (empty) wait(); result = queue[start]; start= (start + 1) % size; empty = (start == end) ? true; notifyAll(); return result; } public synchronized void enqueue(int value) { while (start == end && !empty) //while full wait(); queue[end] = value; end = (end + 1) % size; empty=false; notifyAll(); } }

67 Code Transformation class Queue { invariant: (start in {0,..,size-1}, end in {0,.., size-1}) && (start != end => empty == false) recovery actions: {this = new Queue(); empty = false;} private int[] queue; private int start, end, size; private boolean empty; public Queue(int size){ queue = new int[size]; atomic{ start = 0; end =0; empty= true;} } public synchronized int get() { int result; while (empty) wait(); result = queue[start]; atomic{ start= (start + 1) % size; empty = (start == end) ? true;} notifyAll(); return result; }

68 Code Transformation Cont. public synchronized void put(int value) { while (start == end && !empty) //while full wait(); queue[end] = value; atomic{ end = (end + 1) % size; empty=false;} notifyAll(); } }

69 External Monitoring The possible invariant is: (start in {0,..,size-1}, end in {0,.., size-1}) && (start != end => empty = false) && (start = end && empty = false => producer.wait()) && (empty = true =>consumer.wait()) && [(producer.wait() && !consumer.wait()) || (!producer.wait() && consumer.wait())] New recovery actions: interrupt producer/consumer and initialize it.

70 Recovery Oriented Programming: Summary Programming with self-stabilization enforcing. Eventually safe execution.

71 Talk Conclusions Self-Stabilization as an effective paradigm for creating robust systems. Rigorous approach for designing basic system components Microprocessor Operating system Compiler Evolving and Recovery Oriented

72


Download ppt "Self-Stabilizing Systems as a Base for Autonomic Computing Shlomi Dolev Yinnon Haviv, Reuven Yagel, Olga Brukman."

Similar presentations


Ads by Google