(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Real-Time Design Guidelines and Rules of Thumb Embedded Systems Conference Boston 2011 ESC-309 Dave Stewart, PhD Director of Research and Systems Integration InHand Electronics

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Class Objective To provide some guidelines and rules of thumb to quickly make the right decisions when designing and implementing real-time systems, even if you dont have an in-depth theoretical or practical real-time systems experience. Guideline: Generally a good practice to follow Rule of Thumb: quick decision or approximation Caution: Common Pitfall to Avoid

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September I. Choosing the Right Environment II. Software Decomposition III. Practical Real-Time Scheduling IV. Interprocess Communication V. Implementation Solutions Contents

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Hard vs. soft real-time Predictable systems Preemption Operating systems vs. executives I. Choosing the Right Environment

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Terminology Hard Real-Time Timing requirements must be met precisely Failure to meet requirements leads to significant failure catastrophic (system is destroyed) damage to environment or injury to people economic (product fails in the market place) Soft Real-Time There is some flexibility in the timing constraints

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Hard or Soft? Are these applications Hard or Soft real-time? Flight Control System Bottling Plant Production Line Anti-Lock Braking System DVD Playback Airline Reservation System Internet Video Cellular Phone RF Reception

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Hard or Soft? Are these applications Hard or Soft real-time? Flight Control SystemSoft Bottling Plant Production LineHard Anti-Lock Braking SystemSoft DVD PlaybackHard Airline Reservation SystemSoft Internet VideoSoft Cellular Phone RF ReceptionHard My Opinion Any Surprises?

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Soft Real-Time as a Spectrum Almost all applications can be soft real-time Sometimes, one or more timing constraints are hard Right-most end of spectrum Rarely are all timing constraints hard Few applications are non-real-time Left-most end of spectrum HardNot RT Soft Computer Simulation User Interface Electronic Engine Wireless Communication Internet Video Temperature Control Flight Control

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Real vs. Artificial Constraints Real There is no flexibility in the timing constraint. If it is changed or relaxed, the system will fail. Eg. 150 msec window for filling bottle on conveyor belt. Artificial Timing constraint was determined somewhat arbitrarily. E.g. 40 Hz rate for cruise control. Important to distinguish between Real and Artificial: Artificial constraints can be refined, possibly improving other design factors (like cost & functionality) Real constraints must be met at any cost.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Terminology Predictable The system operates within known bounds. Exact order of operations is not always known. Deterministic A special (extreme) case of predictable. The system operates with exact order of operations known at all times. A Real-Time Systems timing behavior needs to be Predictable, but seldom Deterministic

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Example Dice: Predictable or Deterministic?

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September How do we make a System Predictable? Maximize periodic activities Minimize aperiodic (random) activities Avoid unnecessary preemption Choose appropriate scheduling strategies Eliminate interrupts Use deferrable or sporadic server for aperiodic tasks Use only guaranteed soft real-time tasks Measure execution time to monitor resource usage Implement proper timing error detection and handling

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Terminology Periodic Code that executes at regular intervals Rate refers to how often task executes: E.g. 50 Hz (50 times per second) Period = 1/Rate. E.g. 1/(50Hz) = 20 msec. Aperiodic Code that executes irregularly or randomly. Minimum inter-arrival time is shortest amount of time between two successive aperiodic events.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Periodic In Real-Life, is it easier to schedule? 1hr meeting at the same time each day? 1hr meeting at the same time each week? 4hr meeting at the same time each week? 1hr meeting twice a week, but time changes? Real-Time is like real-life. It is easier to schedule: Tasks that are periodic instead of irregular Less frequent tasks Shorter tasks

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September KISS Principle From the Jargon Dictionary: KISS Principle /kis' prin'si-pl/ n. Keep it short and simple Periodic Tasks help KISS. Execute each task as slow as possible Execute each task for as little time as possible Anything else adds complexity

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Preemption Do you need it? Is it helping or hurting your application?

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Preemption: When it Helps Wide variety of periods and execution times If freq of fastest task < 100 * freq of slowest task, then the task set is likely schedulable without preemption Multiple applications sharing the processor RT and non-RT tasks sharing a processor Hard and soft RT tasks sharing the processor More tolerant to design-time mistakes But this leads to laziness at design time

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Preemption: What it Costs Reduces predictability Increases overhead (scheduling, context switch) Possible race conditions Introduces possibility of priority inversion Significant increase in overhead to avoid race conditions and priority inversion Reduces schedulable bound if blocking is allowed More difficult to test and debug Potential for deadlock if blocking occurs Estimate at least 25% increase in CPU usage for a preemptive system, compared to non-preemptive.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Preemption: Do You Need It? Use KISS to decide: If your application needs it, preemption can be a tremendous benefit to reduce complexity. If you dont need it but use it, your application will be much more complex than necessary. The best compromise might be limited preemption. more later

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Real-Time Operating Systems (RTOS) Every RTOS provides preemption. Why dont any provide the option of non-preemption?

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Non-Preemptive RTOS Called Real-Time Executive (RTE) There are no COTS RTEs because they can be implemented in 5 to 10 lines of code. You cant sell 10 lines of code for $3000. I dont need preemption, so I dont need an RTOS? Maybe I need preemption, so I need an RTOS? Yes COTS: Commercial Off-The-Shelf

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September COTS RTOS When do I use an RTOS? You determine preemption is necessary Lots of COTS input/output devices, like VGA monitor, USB serial ports, touchscreen, ethernet, etc. You are using advanced CASE tools to design and implement software CASE: Computer-Aided Software Engineering

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September COTS RTOS Roll my own RTOS vs. COTS RTOS If you dont have a degree in RTOS design, always use COTS Roll your own only if you are an expert in the field, and there are extenuating circumstances requiring it. E.g. Using RT mechanisms not available in commercial RTOS Domain-specific certifications require 100% control of source code If you roll your own, use a free RTOS as a starting point (eg uCOS, RTLinux) You are in the business of selling RTOS technology Licensing costs: Must compare cost of license(s) to cost of labor. More often than not, cost of labor is substantially higher

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Real-Time Executive When do I use a Real-Time Executive? You determine preemption is not necessary Most or all of the I/O is custom You need 100% control of all code on the processor for software certification Extreme processor resource constraints RAM or ROM < 10 KBytes CPU speed < 10 MHz

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Real-Time Executive Roll my own? Yes, because there are no commercial ones Use a Commercial or Free IDE that provides the compiler, standard libraries, and debugging tools Do NOT hard-code the process flow. Thats not an RTE, its just hacked code. An expert should design the core RTE engine, so that user tasks simply plug in to the framework. IDE: Integrated Development Environment

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September II. Software Decomposition Circular Dependencies are BAD Use a Data Flow Architecture Model of a Real-Time Task

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Circular Dependencies are BAD Circular Dependencies are the primary cause of spaghetti code Use Dependency Trees to spot and remove circular dependencies Eliminating circular dependencies provides the following benefits: Component reusability Fault confinement Incremental testing Modularized performance and timing analysis

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September abc defghi jklstu mnouvwpqr xyz jkl uvwpqr xyz Dependency Tree Example A circular dependency changes the tree into a graph. Avoid circular dependencies at (almost) all costs. abc defghi jklstu mnouvwpqr xyz ghi jklstu mnouvwpqr xyz abc defghi jklstu mnouvwpqr xyz abc defghi jklstu mnouvwpqr xyz

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Avoiding circular dependencies Never have a single globals.h or equivalent file with all header information. Instead, one.h file per module Only put exported declarations in the.h file. Data structures that are private should be defined in the.c file, not in the.h file. If a module is dependent on another, only then should it include the.h file of the other module.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Breaking Circular Dependencies ghi jkl ghijkl shared replacewith This is not always easy. It will force careful reconsideration of the design. Re-designing to satisfy dependency requirements will result in software that is substantially easier to maintain. Additional details is beyond the scope of this class.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Use a Data-Flow Architecture Every task is independent Use the dependency tree to verify this Only use explicit Interprocess Communication to exchange data between tasks task A ipc task A task B task A task B BadGood

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Use a Data-Flow Architecture Independent tasks enable use of real-time scheduling Without real-time scheduling, timing constraints cannot be guaranteed

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Data Flow Architecture Example (Cruise Control System) engine force to brake brake pedal accel accelerator pedal ccui cruise control user interface wheels force to dashboard speedometer position wheel position resolver odometer dx/dt control split-pm f ra f rb x z + – = + – f d f de f dw wheels engine v r U r v z Start with outputs then produce diagram right to left.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Model of a Real-Time Task : IPC Output y 1 y m r 1 r p real-time task : : IPC Input x 1 x m Configuration Constants Sensor Inputs and Control Outputs k 1 k p :

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Model of a Real-Time Task Each task has a main loop that does the following: Read Inputs/Events Do Processing Write Outputs Wait for Event For periodic tasks, event is time-based. For other tasks, event could be an interrupt, message arrival, semaphore wakeup, or any other signal.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Proper Implementation of Periodic Tasks DO (use a framework and wait until to ensure no clock skew) nextstart = clock(); while (1) { nextstart += period; wait until period // wait for timer event call task->cycle() } DONT (implement timing inside task, and use sleep that causes clock skew) taskMain() { laststart = clock(); while (1) { delta = period – (clock() – lasttime); sleep(delta); // wait for timer event do stuff } Additional examples when real-time scheduling examples are presented.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Proper Implementation of Aperiodic Tasks Use Aperiodic Servers More details near end of this talk.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Your Life is a Real-Time System Cyclic executives Rate monotonic algorithm Dynamic scheduling III. Practical Real-Time Scheduling

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Practical Real-Time Scheduling Now: Hard Real-Time Periodic Tasks Non-Preemptive Preemptive Fixed Priority Dynamic Priority Later: Interrupts Aperiodic servers Soft real-time tasks Limited preemption

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Your Life IS a Real-Time System

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Real-Life Scheduling If you have 10 tasks to do, what order to you do them in? If you go to the bank, in what order are you served? If you need to catch a flight at 5pm, how does that affect your schedule? If it is lunch time and you have a report due at 3pm, do you stop for lunch? If the report is due at 7pm, do you stop for lunch? These same kinds of decisions must be made for each task in a real-time system. Thats real-time scheduling.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Basic Cyclic Executive Everything executes at same rate Example: 3 Tasks Each task executes every 20 msec (T A =T B =T C =20) Note: T i =Period of Task i Execution time constraints: C i < T A Note: C i =Worst Case Execution Time of Task i Suitable for simple control loops. A B C ABC time (msec) ABCABCABC

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Basic Cyclic Executive Implementation typedef void (*funcptr)(void); void A(void) { // B and C defined similarly //code goes here } funcptr cyclic[] = { A, B, C}; const int cycliclength = sizeof(cyclic)/sizeof(funcptr); cyclicexec(funcptr cyclic[],int cycliclength, int cycletime) { int i; intrSetPeriod(20msec); while (1) { intrWait(); for (i=0;i<cycliclength;++i) cyclic[i](); } ABC time (msec) ABCABCABC See paper for details on implementing the timer. A B C

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Multi-Rate Cyclic Executive Tasks execute at different rates Example: 3 Tasks A executes every 20 msec (T A =20) B and C execute every 40 msec (T B =T C =40) Execution time constraints: C A +C B < T A && C A +C C < T A time (msec) ABACABAC

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Multi-Rate Cyclic Executive Implementation typedef void (*funcptr)(void); void A(void) { // B and C defined similarly //code goes here } funcptr cyclic[] = { A, B, NULL, A, C, NULL}; const int cycliclength = sizeof(cyclic)/sizeof(funcptr); cyclicexec(funcptr cyclic[],int cycliclength, int cycletime) { int i; intrSetPeriod(20msec); while (1) { for (i=0;i<cycliclength;++i) { if (cyclic[i]==NULL) intrWait(); else cyclic[i](); } time (msec) ABACABAC

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Multi-Rate Cyclic Executive Rates for each task are constrained: Periods of each task must be harmonic To setup schedule, place fastest task in schedule first Then place next fastest, and so on. Execution time is constrained: C i for tasks executing each cycle must be less than period of interrupt.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Harmonic Task Set Period of Task K is multiple of every faster task k>1, T k =n k *T k-1, k, n k are integers Examples of Harmonic: 10, 20, 40, 80, , 30, 120, , 100, 2000 Examples of Non-Harmonic: 10, 25, 50, 75 10, 30, 100, , 120, 2200

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Dynamic Multi-Rate Cyclic Executive Relaxes constraints on period and execution time Uses Earliest-Deadline First (EDF) Algorithm Task with earliest deadline executes next Deadline of current cycle is time task is ready for next cycle Rate/Period constraints: none Execution Time constraints: U i ) <1 (i.e. CPU not overloaded) Note: U i =C i /T i =Utilization of Task I C i for each task is < T 1 /2 i.e. period of fastest task dictates max execution time for every other task Cost of using this method: Increased RTE complexity Requires more thorough analysis to guarantee scheduling feasibility

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Implementation of Dynamic Multi-Rate Cyclic Executive while (1) { do { etimeClock(&now); // Move newly awoken tasks from PauseQ to ReadyQ while (!taskEmptyQ(pauseQ) && etimeCompare(taskSchedTime(pauseQ),now) <= 0) { taskptr=taskDequeue(&pauseQ); etimeAdd(taskptr>schedtime, taskptr->period); taskEnqueue(&readyQ,taskptr); } } while (readyQ.next == NULL); // Head of readyQ has earliest-deadline task taskDequeue(&readyQ, taskptr); taskptr->func(args); taskEnqueue(&pauseQ, taskptr); } (note, some detail has been omitted)

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Preemptive Scheduling Always execute task with highest priority, even if it means preempting a task with lower priority.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September How are Priorities Assigned? Fixed Priority Priorities assigned prior to execution Priorities are Constant E.g. Rate Monotonic Algorithm (RMA) Dynamic Priority Priorities change as a function of time E.g. Earliest-Deadline-First Algorithm (EDF) E.g. Minimum-Laxity-First Algorithm (MLF) Mixed Priority Priorities include both a fixed part and dynamic part E.g. Maximum-Urgency-First Algorithm (MUF)

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Terminology Transient Overload: The state of a real-time system when at least one task will fail to meet its timing constraints because of a lack of resources. Guaranteed Task: A task is said to be guaranteed if it will always meet its deadlines, even in a transient overload situation. Hard real-time tasks must be guaranteed tasks. Task Set: The set of all tasks sharing a processor. Critical Task Set: The set of guaranteed tasks (a subset of the task set). Schedulable Task Set All tasks in the task set are guaranteed tasks. No task ever misses a deadline.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Preemptive Scheduling Example Consider a very simple system with two tasks: P 1 : T 1 = 50msec, C 1 =25msec, U 1 =50%. P 2 : T 2 =100msec, C 2 =40msec, U 2 =40%. U 1 +U 2 =90%. Since utilization is less than 100%, we would expect that both tasks can be scheduled without either missing deadlines. Using Fixed-Priority Scheduling, only two scenarios: Case 1: Priority P 1 > Priority P 2 Case 2: Priority P 1 < Priority P 2 missed deadline: a timing error where a task does not complete its execution in time.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Case 1 Priority P1 > Priority P2 P1: T1= 50msec, C1=25msec, U1=50%. P2: T2=100msec, C2=40msec, U2=40%. P1 P P1 meets 1st deadline 1st deadline P1 P1 meets 2nd deadline P2 meets 1st deadline 2nd deadline P1 1st deadline P2

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Case 2 Priority P1 < Priority P2 P1: T1= 50msec, C1=25msec, U1=50%. P2: T2=100msec, C2=40msec, U2=40%. P1 P st deadline P1 P1 meets 2nd deadline P2 meets 1st deadline 2nd deadline P1 1st deadline P2 P1 meets 1st deadline P1 misses 1st deadline P1 restarts

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Rate Monotonic Algorithm (RMA) An algorithm for assigning fixed priorities The higher the frequency (or shorter the period) of a task, the higher its priority. Case 1 was an example of using RMA assignments. Notice how Case 1 resulted in all tasks meeting their deadlines. Case 2 resulted in some one of the two tasks missing deadlines.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Rate Monotonic Algorithm (RMA) Advantages Optimal fixed priority scheduling algorithm If a task set cannot be scheduled using RMA, then it cannot be scheduled using any other fixed priority assignment. If there is a transient overload, it is guaranteed that the lowest priority tasks not in the critical task set are the ones that can fail. Disadvantages Schedulable bound often less than 100% if task set is not harmonic Schedulable bound is the maximum utilization of the critical task set. Example: See Case 3

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Case 3: Like Case 1 but C 2 & T 2 Change; U 2 Unchanged Priority P1 > Priority P2 P1: T1= 50msec, C1=25msec, U1=50%. P2: T2= 75msec, C2=35msec, U2=40%. P1 P2 P1 meets 2nd deadline P st deadline 1st deadline P1 1st deadline P2 2nd deadline P2 misses 1st deadline

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Rate Monotonic Algorithm (RMA) Even though Utilization only 90% RMA cannot schedule this task set RMA Rules of thumb: To achieve 100% schedulable utilization using RMA, task set must be harmonic. If task set is not harmonic, schedulable utilization can be as low as 70%. To Achieve 100% Utilization with non-harmonic task sets, must use Dynamic Scheduling For example, see Case 4.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Deadline Monotonic Algorithm A Variation of RMA, except that deadline of a task can be earlier than start of next period. Designed to reduce jitter of executing tasks. Problems with Deadline Monotonic Algorithm: Schedulable bound lower than RMA In my experience 99% of the time, jitter is not an issue. Dont use it!

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Case 4: Like Case 3 but use EDF Scheduling Priority P1, P2 = F(deadline) P1: T1= 50msec, C1=25msec, U1=50%. P2: T2= 75msec, C2=35msec, U2=40%. P1 P P1 meets 1st deadline P2 meets 1st deadline P1 1st deadline P2 P1 meets 2nd deadline P1 P2 meets 2nd deadline

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Earliest Deadline First Advantages Schedulable bound is always 100%. All tasks are in critical set if utilization < 100% Optimal dynamic priority scheduling algorithm If a task set cannot be scheduled using EDF, then it cannot be scheduled using any other algorithm. Disadvantages If there is a transient overload, there is no control over which task fails. None of the tasks are in critical set if utilization >= 100%

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Mixed Priority Scheduling Combines the Advantages of RMA and EDF Schedulable bound of 100% for critical task set In case of transient overload, critical task set can still be guaranteed Various Options to Mixed-Priority Scheduling E.g. Maximum-Urgency-First (MUF) E.g. Limited Preemption Rate Groups Mixed-priority scheduling is a more advanced topic; more discussion later.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Baseline Scheduling What Method Should You Use? Non-Preemption Basic cyclic executive Multi-rate cyclic executive Dynamic cyclic executive Preemption Rate monotonic algorithm Earliest-deadline-first algorithm More complex mixed-priority algorithm Answer: KISS Principle Pick the simplest one that meets your needs.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Other Considerations Other design needs may alter the choice of a basic scheduling method. For example: Interprocess communication Aperiodic events Soft real-time support Interaction with non-real-time systems Dynamic system configuration changes Overloaded resources I/O requirements These and other issues discussed next.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September IV. Interprocess Communication Race conditions Semaphores Priority inversion Real-time IPC mechanisms

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Interprocess Communication Real-Time Applications are built as collections of tasks executing at one or more rates. Tasks exchange data through IPC. If done incorrectly, IPC is one of the primary sources of problems in RT Systems.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Non-Preemptive Environments Use Shared Memory between tasks. Fastest form of IPC No RTE overhead to access shared data No concern of race conditions since tasks accessing the shared data cannot be preempted. If there are interrupts, they can preempt non-preemptive tasks. Must be treated as a preemptive environment from an IPC perspective, if the interrupt handler shares memory or resources with any of the tasks.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September IPC in Preemptive Environments Shared Memory and Resources Cause Problems. This includes: Global variables Static local variables Shared memory segments I/O device registers Library functions that are not Multi-Tasking Safe

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Race Conditions The main problem with sharing in preemptive environments is ensuring data does not get corrupted due to a race condition. The One Lane Bridge is Shared. It is an analogy for a shared variable. If Trains do not take special precautions, an accident will occur. If software doesnt take precautions, data corruption will occur. This is a Race Condition.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Semaphores A Semaphore is a traffic light The One Lane Bridge is Shared. It is an analogy for a shared variable. If Trains do not take special precautions, an accident will occur. If software doesnt take precautions, data corruption will occur. This is a Race Condition.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Semaphores Semaphores are a common solution in operating systems to deal with race conditions. In real-time systems they lead to priority inversion. In fact, any IPC mechanism that involves blocking, including monitors, condition variables, and message passing, can lead to priority inversion.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Priority Inversion Blocking of a high priority task due to a lower priority task locking a shared resource.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Priority Inversion Shown by Example Assume 3 Tasks: PH: A high priority task PM: A medium priority task PL: A low priority task

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Priority Inversion Shown by Example

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Solutions to Priority Inversion There are two fundamental ways to deal with race conditions while avoiding priority inversion: Use a mechanism that adjusts priorities of low-priority tasks holding resources. Use a non-blocking mechanism. WARNING! Many RTOS provide blocking IPC mechanisms that do not adjust priorities. Do NOT use these mechanisms in Real-Time Systems!

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Adjusting Priorities: Priority Inheritance Protocol Warning: For demonstration only. Not a good protocol.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Adjusting Priorities: Priority Inheritance Protocol PL starts PL gets R1 PH starts PH needs R1 PM starts PL releases R1 PH releases R1 PH ends PM ends PL ends PH PM PL ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ due to priority inversion PL assumes priority of PH PM does not execute, because priority of PL is temporarily higher holding R1 suspended ready to run running blocked, waiting for R1 state of task ^^^^^

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Adjusting Priorities Priority Ceiling Protocol (PCP) Highest Locking Protocol (HLP) is very similar. Similar to Priority Inheritance, except that each resource is given a priority ceiling, which is the priority of the highest-priority task that may lock the resource. When a resource is locked, task immediately inherits the priority ceiling. Advantage: It Works Disadvantages: Significant RTOS overhead Significant reduction in schedulable bound due to blocking Many RTOS do not provide this mechanism

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Non-Blocking RT IPC The alternative to PCP is to use a non-blocking form of communication. Some examples: State variable mechanism Non-blocking message passing Triple-buffer mechanism FIFOs

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September State Variable (SVAR) Communication Different versions for different environments Used in state-based design to store states. E.g. velocity of a vehicle can be a state variable in a cruise control system, because only the current speed is important. Key attribute of states: Only the most recent data is needed. Allows tasks executing at different rates to communicate with each other: If a producer is faster than a consumer, then the consumer may miss some variable changes. If a producer is slower than a consumer, then the consumer may read the same value multiple times. State data is small, so that a task can lock out interrupts (thus preventing preemption) for a few microseconds to update the state. Each module reads the state and makes a local copy before executing each cycle, then updates the state after completion of its cycle. It does not access the state variables at any other time.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September SVAR Architecture Non-Preemptive Environment Global Table Intr A 1 task A 2 task A k

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September SVAR Architecture Non-Preemptive with Interrupts Global Table local table Intr A 1 task A 2 task A k

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September SVAR Architecture Preemptive Single Processor Global Table local table local table local table task A 1 task A 2 task A k

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Example Consider following Data-Flow

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September SVAR Global and Local Tables Contain Data as Follows x d tballcinterprmmsgfwdkinginvkin x r. x d x m x r x r N DOF r r m DH N DOF m m N DOF DH x m DH N DOF DH x m x r v r x d r m m Global Table: Local Tables: PBOs:

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September SVAR Architecture Multiprocessor Shared Memory Environment Global Table local table task A 1 Task A j Processor A local table local table Task K 1 k j Processor L local table memcpy() or DMA

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September SVAR Architecture Network Environment Global Table local table Task K 1 k j Processor L local table network messages Processor X data server data client local table Task K 1 k j Processor A local table data client

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Non-Blocking Message Passing Just Like Regular Message Passing, Except: Sending: if message queue is full, abort operation and continue. Receiving: if message queue is empty, return immediately without data. In such cases, it is important to design the system to properly handle the above cases.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Messages vs. States In most cases, messages can be converted to states, and vice versa. Messages are best used for real-time client/server architectures. States are best used in feedback control applications.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September SEND (apply brake)... control module apply brake brake = ON... IF brake == ON activate brake ELSE brake 2 brake ON global table (shared memory) brake ON brake ON local table... IF brake == ON activate brake ELSE brake 1 brake ON local table... Using Messages, Apply Brake: Using States, The Brakes Should be ON: control module RECEIVE (&message) IF message == apply brake activate brake ELSE : brake 2 RECEIVE (&message) IF message == apply brake activate brake ELSE : brake 1

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Triple Buffer Mechanism Used for point-to-point state communication Does not require any hardware or RTOS support Three (or more) copies of the data exist. No two tasks ever read and write from the same copy at the same time. When a writer finishes, it marks the buffer as new, and moves on to a different buffer. Always at least one buffer not being used by other tasks. This eliminates the need to tightly synchronize readers and writers.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Triple Buffer Mechanism

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September FIFOs (First-In-First-Out) Similar to triple-buffer mechanism, except that it guarantees all data is available. Similar to message passing, but with less overhead and suitable for large volume of data. On average, consumer must be at least as fast as producer, otherwise data loss can occur. Can be used for communication between real-time and non-real-time tasks.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Other RT IPC There are hundreds (or thousands) of real-time IPC mechanisms described in the literature. Most RTOS only implement zero to one or two of them. You may find you need to implement your own. Consider IPC in conjunction with scheduling. If scheduling can prevent race conditions, then you dont need complex IPC. Instead, you can just use simple shared memory.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September V. Implementation Solutions Timing failure detection and handling Soft real-time scheduling Interrupts and aperiodic events Dynamic system configuration changes Timing anomalies

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Timing Failure Detection and Handling Examples of Timing Errors: Missed deadline Using too much execution time End-to-end response time too long Incorrect sampling or control rate Clock skew Too much jitter Inaccurate system or real-time clock

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Online Timing Error Detection and Handling An RTOS, RT Executive, or component framework may include built-in mechanisms to detect some timing errors. Example to follow considers missed deadline and using too much execution time. Used as the basis for implementing guaranteed soft real-time tasks and aperiodic servers.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Timing Failure Handler (TFH) Implementation Requires a Spare Timer Each time a new task is being swapped in, scheduler programs timer with lesser of the following two values: tasks deadline; set type to DEADLINE estimated remaining execution time; set type to MAXEXEC Interrupt handler for this timer is defined as follows: (see next slide)

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September TFH Implementation tfhhandler(int type) { switch (type){ case DEADLINE: /* missed deadline */ set_priority(Ph); nextstart += Tds; set_deadline(nextstart,Cds,Tds); return; case MAXEXEC: /* execution time used up */ set_priority(Pl); return; } /* end switch */ } D.B. Stewart and P.K. Khosla, Mechanisms for Detecting and Handling Timing Errors, Communications of the ACM, v.40, n.1, pp , January

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September TFH Implementation If DEADLINE error Ensure task has its normal priority. This task will continue executing in time allotted for its next period. Thus it now has a new deadline at end of next period. Task will be forced to skip its next period to reclaim the time, resulting in a missed cycle. If MAXEXEC error The amount of reserved execution time by this task has been used up. Lower priority of task, so that other high priority tasks can execute. If there is leftover CPU time, then this task will execute. When it is done, its priority gets raised back to normal on next cycle. If there is insufficient CPU time, a DEADLINE error will follow.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Timing Failure Handler Uses Guaranteed Soft Real-Time Tasks Use the TFH to detect missed timing constraints, then update the tasks accordingly Aperiodic Servers Use the TFH to detect aperiodic events that use more than budgeted capacity, and use that to manage priorities and schedule Above items are described next.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Soft Real-Time Scheduling Soft real-time systems are always one of the following two: Accidental Soft Real-Time Guaranteed Soft Real-Time Most systems claimed to be soft real-time are accidental soft real-time

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Accidental Soft Real-Time These are systems that are claimed to be soft real-time, but in reality they do not provide any real-time guarantees. Typical features of such systems: System can usually keep up to real-time. If the system fails to meet its timing requirement, the system does not crash, but the result still appears to meet requirements. Tried to get hard real-time, but failed. Possible starvation of the soft real-time tasks.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Guaranteed Soft Real-Time It is acceptable for the primary timing constraints to not always be met. There is a secondary set of timing requirements that are always met. Guarantee of no starvation of any soft real-time task. Actively modify scheduling to provide degraded performance during transient overloads.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Using TFH to Implement Guaranteed Soft Real-Time Tasks Suppose a task has the following execution profile: 100% 50% 0% 0exec time WCET=40msec. 80% of the time, task uses 26 msec or less. 80%

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Using TFH to Implement Guaranteed Soft Real-Time Tasks Suppose a task has the execution profile on previous slide: If task is hard real-time, then 40 msec must be reserved. If task has a period of 60 msec, then utilization for the task U=40/60=66.7%. If task is soft real-time, then set its capacity to 26 msec. Then U=26/60=43%. If execution takes less than 26 msec (as happens 80% of the time), then task will meet its deadline. If execution takes more than 26 msec, MAXEXEC error is generated. Task gets lower priority for any execution time greater than 26 msec until its deadline time is reached. If task misses deadline, then in worst case it has 14 more msec to execute, during its next period. It is guaranteed to meet its next cycle, and thus will never miss two deadlines in a row. The net effect of such guaranteed soft real-time tasks is that in the worst-case, when the CPU is fully loaded, the task will execute at half rate.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Interrupts Since interrupts cannot be scheduled, they always have highest priority. This could lead to priority inversion. Periodic interrupts (like a regular timer interrupt) can be treated in RT analysis as the highest priority periodic tasks. WCET must be known. To maximize schedulable utilization, the period must be less than the period of the fastest task in the system. Randomly arriving interrupts can create undesired effects. If they are necessary, keep execution as small as possible. For longer execution, use an aperiodic server, with the interrupt handler sending a signal to trigger execution of the server. Well-behaved interrupts only come occasionally, with a known MIAT. In analysis, randomly arriving interrupts are defined to have period T=minimum inter-arrival time. WCET must be known. A predictable system cannot be built if interrupts do not have a minimum inter-arrival time. Must use aperiodic servers instead. MIAT = minimal inter-arrival time

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Aperiodic Servers Aperiodic servers offer the ability to predictably deal with randomly arriving events. Predictability is achieved by reserving some CPU time to handle aperiodic events. TFH is used to managed that reserved CPU time. A Deferrable Server uses the same TFH Handler as soft real-time tasks. Sporadic Server is more complex to implement. Not much of an improvement over Deferrable Server. D.B. Stewart and P.K. Khosla, Mechanisms for Detecting and Handling Timing Errors, Communications of the ACM, v.40, n.1, pp , January

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Aperiodic Server Implementation An aperiodic server can be implemented just like a soft real-time periodic task. Period T = Replenishment Time WCET C = Capacity Worst-case utilization U=C/T. Depending on the objectives of the aperiodic server, there are different ways to define T and C.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Aperiodic Servers Types Hard Real-Time requiring fastest response time possible. MIAT and WCET must be known. Period T aperiodic =MIAT/K; C=WCET/K. K is selected such that T aperiodic < T fastest_task. Disadvantage: U=C/T could be significant, as this task will have highest priority in the system. If possible, design system to relax the fastest response time possible.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Aperiodic Servers Types Hard real-time, with deadline. Immediate response is not needed, but execution must complete by a given deadline. Set T aperiodic =MIAT, C=WCET. U=C/ T aperiodic is same as previous case and could still be significant, but this server might have lower priority than some other hard real-time tasks thus improving overall schedulability.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Aperiodic Servers Types Soft real-time. If MIAT not known or WCET not known, then aperiodic server must be soft real-time. Decide what percentage of CPU to reserve for processing the aperiodic event. Thus U defined first. Set T aperiodic =desired response time. Set C=U* T aperiodic. If event arrives, then aperiodic server will attempt to process it within the desired response time. If unable, it will continue processing as soon as possible, but not using more than the reserved amount of CPU time (unless no other real-time tasks with reserved execution time are ready to run).

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Aperiodic Servers Types Soft real-time. Priority of the aperiodic server can be controlled: To increase priority: decrease both T and C by a factor of K (K>1). To decrease priority: increase both T and C by a factor of K (K>1). If desired response time rarely being achieved, then increase U, the reserved CPU utilization for the server.

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Dynamic System Configuration Changes Many real-time systems are implemented with multiple modes of operation Each mode might execute a different set of tasks Create a finite state machine (FSM) to define modes as states, and mode changes as transitions. Only change set of executing tasks when mode changes. Thus each state can be defined to have a specific task set. Analyze RT performance for each state. Changing the period of a task is equivalent to a mode change. Use dynamically reconfigurable software to change states in real- time. As one example, see D.B. Stewart, Software Components for Real- Time Systems, Embedded Systems Programming, December

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Summary - Objective To provide some guidelines and rules of thumb to quickly make the right decisions when designing and implementing real-time systems, even if you dont have an in-depth theoretical or practical real-time systems experience. Guideline: Generally a good practice to follow Rule of Thumb: quick decision or approximation Caution: Common Pitfall to Avoid

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Summary - Content I. Choosing the Right Environment II. Software Decomposition III. Practical Real-Time Scheduling IV. Interprocess Communication V. Common Problems and Solutions

(c) 2011 Dave Stewart - InHand Electronics - Systems Conference Boston, September Additional Resources Publications on Embedded Research Solutions Website Embedded Systems Conference Proceedings Embedded Systems Programming Magazine Archive Real-Time Systems Symposium (RTSS) [for theory] Requires IEEE Computer Society membership for access EG3 Embedded Systems Portal InHand Electronics