Implementing Fault Tolerant Systems with Windows CE.NET Reliable System Design 2010 by: Amir M. Rahmani

matlab1.ir CONTENTS Embedded System Real-Time System Windows CE.NET Definitions and Concepts Software Fault Tolerance Techniques Key Operating System Properties Applying Fault Tolerant Techniques Conclusions

matlab1.ir What is an Embedded System? Electronic devices that incorporate a computer (usually a microprocessor) within their implementation. A computer is used in such devices primarily as a means to simplify the system design and to provide flexibility. Often the user of the device is not even aware that a computer is present.

matlab1.ir Embedded Device Aerospace: space exploration (e.g., the Mars Pathfinder) Automotive : air bag controls, GPS mapping Communications : Satellites; network routers Computer Peripherals : Printers, scanners Home : Dishwashers, microwave ovens Industrial : Elevator controls, robots Medical : Imaging systems (e.g., XRAY, MRI(

matlab1.ir What is a Real-Time System? Real-time systems process events Events occurring on external inputs cause other events to occur as outputs. Minimizing response time is usually a primary objective, otherwise the entire system may fail to operate properly.

matlab1.ir Hard/Soft Real-Time Systems Soft Real-Time System Compute output response as fast as possible, but no specific deadlines that must be met (delay doesn’t cause application fail) ex: ATM Hard Real-Time System Output response must be computed by specified deadline. (delay causes application fail) ex: mussel tracker Fault Tolerant Real-Time Scheduling

matlab1.ir What is Windows CE.NET? Combines an advanced real-time embedded operating system with the most powerful tools for rapidly creating the next generation of smart devices. Windows CE is optimized for devices that have minimal storage. Devices are often configured without disk storage, and may be configured as a "closed" system that does not allow for end user extension.

matlab1.ir Why Windows CE.NET? Very popular embedded OS for PDAs and mobiles Microsoft’s latest 32-bit embedded OS platform Easy to use, familiar development tools Real-Time Communication Broad CPU support Full feature functionality and flexibility Hard real-time preemptive multitasking kernel Easy Platform Development with Platform Builder

matlab1.ir Thin Clients Digital Audio Receivers and Players Smart Displays Voice-over IP Devices Medical Devices Industrial Automation Wide range of devices MobileHandhelds Set-Top Boxes Gateways

matlab1.ir Windows Mobile 2005 Windows CE.NET

matlab1.ir Windows CE: Different Hardware Desktop/Laptop PC 2 GHz Pentium IV 256 K  512 K cache 512 M  1 G DRAM 100 G hard drive 1280x1024 display 50 Watts Keyboard & mouse Extensible through PCI, AGC, USB, EISA, 1394, PC-Card, CF, enet, … Device 400 MHz RISC 4 K  8K cache 4 M  ~32 M DRAM 4 M  ~32 M Flash / ROM 170x170  640x480 <1 to 2 Watts Stylus or Thumb Not very extensible PC-Card, SD, CF

matlab1.ir Definitions and Concepts A System is a collection of interacting components that deliver a service through a service interface to a user. Dependability of a computing system is the ability to deliver service that can justifiably be trusted. The system state is the set of component states. An error is a system state that may lead to failure.

matlab1.ir Definitions and Concepts Availability, the readiness for correct service. Reliability, the continuity of that service. Safety, the avoidance of tragic consequences on the environment. Security, the prevention of unauthorized access. Fault tolerance is the ability of a system to deliver of correct service in the presence of faults.

matlab1.ir Specification A key part of the specification is the identification and classification of faults that must be tolerated or otherwise remediated. The specification should define fault and error scenarios, consider system response when faults occur in coincidence, and consider the likelihood and handling of latent errors.

matlab1.ir Implementation Implementation begins with the partitioning of the system into subsystems. Subsystems implement local error detection and recovery, and provide support for higher- level detection and recovery. Error detection can be performed concurrently with the delivery of the service, or by preemptively suspending the service to execute tests.

matlab1.ir Software Fault Tolerance Techniques Two common software fault tolerance schemes based on redundancy and the assumption that the occurrence of correlated faults are unlikely, are the use of : Recovery Block N-Version Programming

matlab1.ir Recovery Block The Recovery Block (RB) design makes use of a primary and one or more alternate program blocks. Each of which performs the specified operations, but in different ways. A test on the results determines the acceptability of the results.

matlab1.ir Recovery Block CurrentMethod = PrimaryMethod( ); do { result = CurrentMethod(); if (AcceptanceTest(result) == OK) return result; } While(CurrentMethod=NextAlternateMethod()); fail();

matlab1.ir N-Version Programming uses parallel execution of N independently developed functionally equivalent software versions. The outputs of all versions are examined in a voter block to determine the correct output, if one exists. NVP is commonly called voter-based fault tolerance.

matlab1.ir Key Operating System Properties Understanding Windows CE.NET's implementation of each of these areas is critical to develop fault tolerant applications:  - proper use and partitioning of application code into processes and threads  - correct setting of priorities  - use of exception handling  - implementation of watchdog behaviors

matlab1.ir Processes and Threads Each application that runs under Windows CE.NET is called a process. The basic unit of execution is the thread. Threads execute any part of the code in the process. Threads allow the application to perform multiple tasks at the same time. Support of preemptive multitasking allows Windows CE.NET to create the effect of executing multiple threads simultaneously.

matlab1.ir Processes and Threads Windows CE.NET supports up to 32 processes and as many threads as memory permits. Each process has a designated primary thread the WinMain function. The key difference between threads within the same process and threads in two different processes is the memory protection and the context switch times.

matlab1.ir Synchronization Objects Partitioning a monolithic application into several threads and processes requires coordination and synchronization. Some of synchronization objects that enable communication include: critical sections mutexes semaphores events

matlab1.ir Exception Handling Windows CE.NET supports the try-except and try-finally Microsoft extensions to the C language to pass execution control to exception handling code. The compilers included in Windows CE.NET support the C++ try, catch and throw exception handling code.

matlab1.ir Priorities Win CE.NET provides 256 priority levels that you can set on a per thread basis. The choice of which thread priorities to use is very critical. Thread priorities from 0-247, with 0 being the highest priority, are referred to as the real-time priorities and require a call to CeSetThreadPriority() to access them. The normal thread priorities from are accessed by using SetThreadPriority().

matlab1.ir Real-Time Thread Priorities 0-98(-20) : Open – Real Time Above Drivers 99 : Power management Resume Thread : USB OHCI UHCI, Serial : Irsir1, NDIS, Touch : Open – Device Drivers 145 : PS2 Keyboard : Open – Device Drivers : Open – Real Time Below Drivers

matlab1.ir Normal Thread Priorities 248 : Power Management 249 : WaveDev, Mouse, PnP,Power 250 : WaveAPI 251 : Power Manager Battery Thread : Open

matlab1.ir Applying Fault Tolerant Techniques 1- Partition into Threads and Processes 2- Implement Watchdog Threads 3- Implement Exception Handling 4- Implement Interrupt Level Fault Detection

matlab1.ir Partition into Threads and Processes Threads and processes are the building block of Windows CE.NET applications. Isolation of logic into separate threads with the same process is the first level of protection. Moving threads to other processes ensures addition protection. Simple example of a monolithic application is:

matlab1.ir #include "stdafx.h" int g_nCount1 = 0; int g_nCount2 = 0; int WINAPI WinMain(…) { while( 1 ) { Algoritm1(); if( g_nCount1 == 4 ) { Algorithm2(); g_nCount1 = 0; } if( g_nCount2 == 9 ) { Algorithm3(); g_nCount2 = 0; } g_nCount1++; g_nCount2++; Sleep(5); } return 0; }

matlab1.ir Breaks up the monolithic application #include "stdafx.h" HANDLE g_htAlgoritm1; HANDLE g_htAlgoritm2; BOOL g_fFinished = FALSE; DWORD WINAPI ThreadAlgoritm1 (...); DWORD WINAPI ThreadAlgoritm2 (...);

matlab1.ir int WINAPI WinMain(…) { g_htAlgoritm1 = CreateThread(..,ThreadAlgorithm1,...); g_htAlgoritm2 = CreateThread(..,ThreadAlgorithm2,...); if( !g_htAlgoritm1 || !g_htAlgoritm1 ) return 20; if( !CeSetThreadPriority(g_htAlgoritm1,50) ) return 30; if( !CeSetThreadPriority(g_htAlgoritm2,60) ) return 30; while( !g_fFinished ) { Sleep( 500 ); } return 0; } DWORD WINAPI ThreadAlgoritm1( LPVOID lpvParam ) } while( !g_fFinished ) { Sleep( 5); Algorithm1(); } return 0; } DWORD WINAPI ThreadAlgoritm2( LPVOID lpvParam ) }while( !g_fFinished ) { Sleep(25); Algorithm2 (); } return 0; }

matlab1.ir They key elements to Process 1 are: Create the two threads ThreadAlgorithm 1 & ThreadAlgorithm 2 Set the thread priorities: Set Thread Algorithm 1 to 50 (higher than the Algorithm 2) Set Thread Algorithm 2 to 60. Loop every 500 ms waiting for the g_fFinished flag to be set ThreadAlgorithm 1 and ThreadAlgorithm 2 sleep the appropriate amount of time and call their respective algorithms

matlab1.ir #include "stdafx.h" HANDLE g_htAlgoritm3; BOOL g_fFinished = FALSE; DWORD WINAPI ThreadAlgoritm3 (...); Int WINAPI WinMain(...) { g_htAlgoritm3 = CreateThread(…,ThreadAlgorithm3,…); if( !g_htAlgoritm3 ) return 20; if( !CeSetThreadPriority(g_htAlgoritm3, 40 ) ) return 30; while( !g_fFinished ) { Sleep( 500 ); } return 0; } DWORD WINAPI ThreadAlgoritm3 ( LPVOID lpvParam ) } while( !g_fFinished ) { Sleep(50); Algorithm3 (); } return 0; {

matlab1.ir They key elements to Process 2 are: Create the one thread ThreadAlgorithm3 Set thread algorithm 3 to 40 (higher than the Algorithm 1 & 2). Loop every 500 ms waiting for the g_fFinished flag to be set. ThreadAlgorithm3 sleeps the appropriate amount of time and calls Algorithm 3. The two processes have allowed for partitioning of the algorithms and have increased their ability to run in the event of logic faults in the others.

matlab1.ir Partition into Threads and Processes The developer now has a tremendous amount of flexibility. The ability to set the thread priorities allows for the prioritization of the algorithms, which was not available in the monolithic case. The advantages of such a partitioning are just the beginning of the fault tolerant techniques available to the Windows CE.NET developer.

matlab1.ir Implement Watchdog Threads Watchdog threads can be implemented to verify that a particular set of logic is continuously executing. a control thread with the responsibility to set a particular watchdog event for each control cycle. If the watchdog thread ever wakes up without the control thread having run, the watchdog thread will execute a shut down behavior.

matlab1.ir #include "stdafx.h" HANDLE g_hevWatchdog; HANDLE g_htWatchdog; HANDLE g_htControl; BOOL g_fFinished; DWORD WINAPI ThreadWatchdog (…); DWORD WINAPI ThreadControl (...); Simple watchdog example

matlab1.ir int WINAPI WinMain(...) { g_hevWatchdog = CreateEvent(NULL, FALSE, FALSE,NULL); if(!g_hevWatchdog) return 10; g_htWatchdog = CreateThread(NULL, 0, ThreadWatchdog, NULL, CREATE_SUSPENDED, &dwThreadID ( ; if( !g_htWatchdog )return 20; g_htControl = CreateThread(NULL, 0, ThreadControl, NULL, CREATE_SUSPENDED, &dwThreadID ( ; if( !g_htControl ) return 20; if( !CeSetThreadPriority(g_htWatchdog, 5 )) return 30; if( !CeSetThreadPriority(g_htControl, 10 )) return 40; ResumeThread( g_htWatchdog ); ResumeThread( g_htControl ); while( !g_fFinished ) { Sleep( 500 ); } return 0; }

matlab1.ir DWORD WINAPI ThreadControl(LPVOID lpvParam ) { while( !g_fFinished ) { SetEvent( g_hevWatchdog ); // Do application processing here... Sleep( 5); } return 0; } DWORD WINAPI ThreadWatchdog( LPVOID lpvParam ) } while( !g_fFinished ) { result = WaitForSingleObject( g_hevWatchdog, 500 ); if( result == WAIT_TIMEOUT ) { // We have a failure, application thread is dead!!! WRITE_PORT_UCHAR( (PUCHAR)PowerAddress,POWER_SHUTDOWN); } } return 0; }

matlab1.ir Key steps in initializing the controller Create the watchdog event. Set up the watchdog and control thread suspended. Set the watchdog thread priority to 5 (higher than the control thread). Set the control thread priority to 10. Resume both the watchdog and control threads. Wait for the g_fFinished flag to be set.

matlab1.ir Exception Handling Exceptions are anomalous situations that may occur during your program execution. including common software faults such as arithmetic value domain errors, and access to invalid memory addresses. With C++ exception handling you can directly handle exceptions or pass exception control up to a higher level where handling the error is more appropriate.

matlab1.ir #include int WINAPI WinMain(…) { char *buf; try { while( 1) { buf = GetBuffer(); if( buf == 0) throw "Memory Failure from GetBuffer!"; Sleep(4); } } catch( char* str ) { cout << "Exception raised! " << str <<<'\n'; } catch( … ) { throw; // Pass control to the next } return 0; }

matlab1.ir Conclusions Internet and handset devices are growing quickly, so threats to the PDAs and mobiles become more and more serious. The entrance of Windows CE.NET into the embedded community has enabled a new class of complex and robust embedded software products. Partitioning monolithic, complex algorithms into separate threads and processes, proper usage of thread priorities, the use of exception handling, the implementation of watchdog behaviors and algorithmic redundancy are readily available techniques that may be used in a Windows CE.NET implementation to deliver more dependable services.

matlab1.ir References Nat Frampton and Richard Lee, “Implementing Fault Tolerant Systems with Windows CE.NET”, Mike Hall, “Windows CE Development”, Daniel W. Lewis, “Embedded Systems”, 2000.