1. Costly Mistakes of Real-Time Software Development Dave Stewart, PhD Director of Research and Systems Integration InHand Electronics dstewart@inhand.com www.inhand.com ESC 223

2. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Why this presentation? 2 Rookies and Experts make the same mistakes over and over again.

3. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com What Constitutes Costly Mistakes? 3 Initial software effort is over budget Initial software effort is over budget Delivery is late Delivery is late Mistake requires an expensive solution Mistake requires an expensive solution One or more requirements not met One or more requirements not met Quality issues that lead to poor reputation Quality issues that lead to poor reputation High maintenance costs to fix software High maintenance costs to fix software

4. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com The Order is Subjective 4 is costliest mistake … but just in my opinion. Every person may have their own personal order.

5. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com The Order is Not Really Important 5 What is important is that the mistake is on the list! Correcting just ONE mistake can save thousands of dollars or significantly improve quality and robustness of software. Correcting SEVERAL mistakes can lead to savings and improvements that are incalculable!

6. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com No Emulators of Target Applications 6

7. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Use an Emulator to Reduce Costs 7 Prototyping before hardware is available Prototyping before hardware is available Faster development with better tools Faster development with better tools Enable more developers Enable more developers Obtain early customer feedback Obtain early customer feedback Deeper understanding Deeper understanding Mistakes that do happen are safer and cheaper! Mistakes that do happen are safer and cheaper!

8. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Its just a glitch 8 Never assume that a problem has been fixed magically Note problem in your log book immediately! Spend some time to try and fix the problem

9. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com its just a glitch 9 Recognize most likely causes Recognize most likely causes timing error (i.e. real-time scheduling issue) timing error (i.e. real-time scheduling issue) race condition race condition memory corruption memory corruption hardware problem hardware problem Almost any other type of error would be easy to replicate, and would not be a random or rare glitch. Almost any other type of error would be easy to replicate, and would not be a random or rare glitch.

10. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com During Design, Take Precautions 10 Due to high cost of trying to debug glitches, take precautions that will minimize potential issues Due to high cost of trying to debug glitches, take precautions that will minimize potential issues Formal code review Formal code review Minimize shared resources and memory Minimize shared resources and memory Limit use of interrupts Limit use of interrupts Select appropriate real-time scheduling algorithm Select appropriate real-time scheduling algorithm Analyze expected real-time performance Analyze expected real-time performance Use deadlock-free IPC solutions Use deadlock-free IPC solutions

11. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Actively Troubleshoot 11 Knowing that glitches are usually one of a few specific types of errors, actively try to replicate. E.g. Knowing that glitches are usually one of a few specific types of errors, actively try to replicate. E.g. Force context switches inside race conditions (e.g. by adding a sleep command) to validate mutexes Force context switches inside race conditions (e.g. by adding a sleep command) to validate mutexes Intentionally cause an overload of the system to see how it fails if CPU is overloaded Intentionally cause an overload of the system to see how it fails if CPU is overloaded Check for stack or heap corruption Check for stack or heap corruption Monitor progress of software and hardware using a logic analyzer (see ESC Class Troubleshooting Real-Time Software Issues Using a Logic Analyzer) Monitor progress of software and hardware using a logic analyzer (see ESC Class Troubleshooting Real-Time Software Issues Using a Logic Analyzer) Incrementally add debug information so that next time glitch happens, more information is available Incrementally add debug information so that next time glitch happens, more information is available

12. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Poor or No Software Diagrams 12

13. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Poor or No Software Diagrams 13 typedef struct _def_t { struct _def_t *next; struct _def_t *prev; char name[8]; short loval; short hival; } def_t; typedef struct _xyz_t { int i; float f; short s[2]; unsigned char b[8]; } xyz_t; typedef struct _abc_t { def_t *def; xyz_t *xyz; short ndef; } abc_t; Without a corresponding diagram, there is no easy way to know what this code is supposed to do. By reverse engineering it, at best you can determine what the code does, but is that what it is supposed to do? Diagrams represent the design, no diagrams means code was implemented directly from requirements, and will likely not be very robust nor efficient.

14. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Poor or No Software Diagrams 14 typedef struct _def_t { struct _def_t *next; struct _def_t *prev; char name[8]; short loval; short hival; } def_t; typedef struct _xyz_t { int i; float f; short s[2]; unsigned char b[8]; } xyz_t; typedef struct _abc_t { def_t *def; xyz_t *xyz; short ndef; } abc_t; Diagrams make it clear what code is supposed to do. Errors are more easily identified, as code does not reflect the diagram.

15. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Creating Good Diagrams 15 Create a legend for every diagram Create a legend for every diagram Every block, symbol, line, shading, color, and font type should be specified in legend. Every block, symbol, line, shading, color, and font type should be specified in legend. Any deviation from legend shows an error in the design Any deviation from legend shows an error in the design Land Water

16. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Creating Good Diagrams 16 Use Drawing or CASE Tools Use Drawing or CASE Tools E.g. UML (Unified Modeling Language) E.g. UML (Unified Modeling Language) UML is nothing more than a standard set of legends for different types of diagrams UML is nothing more than a standard set of legends for different types of diagrams Use different diagrams to represent multiple views Use different diagrams to represent multiple views Data Flow Data Flow Process Flow Process Flow Sequence Diagrams Sequence Diagrams Timing Diagrams Timing Diagrams State Machines State Machines Use Cases Use Cases Data Structures Data Structures Class Diagrams Class Diagrams Each layer should have Each layer should have Architectural Design (how components are integrated) Architectural Design (how components are integrated) Detailed Design (contents of each component) Detailed Design (contents of each component)

17. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Inadequate Design for Test 17 Most common reason for not testing code is because it is difficult to test Most common reason for not testing code is because it is difficult to test Yet that is precisely when testing is most needed Yet that is precisely when testing is most needed When a system is being designed, every requirement needs to have the following question answered, how will this requirement be tested? When a system is being designed, every requirement needs to have the following question answered, how will this requirement be tested? If there is no answer, then guaranteed this is the requirement that will cause problems If there is no answer, then guaranteed this is the requirement that will cause problems Update the design to include methods to test! Update the design to include methods to test!

18. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Example: Too Many Branches 18 It is extremely difficult to test many branches. It is extremely difficult to test many branches. Such code is usually result of implementing directly from requirements, that tend to have a lot of if conditionals. Such code is usually result of implementing directly from requirements, that tend to have a lot of if conditionals. Solution: change the design. Define the system as a finite state machine that is easier to test. Solution: change the design. Define the system as a finite state machine that is easier to test.

19. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Improve Status Output to Console 19 Bad: Bad: Reading 100 bytes Reading 100 bytes Good: Good: SPI R 100 Cmd=0x12 Data=45 12 20 10 … SPI R 100 Cmd=0x12 Data=45 12 20 10 … Bad: Bad: State is 4 State is 4 Good: Good: [pwr_ctrl.c Line 112 pwr_VoltageSet()] state=4 [pwr_ctrl.c Line 112 pwr_VoltageSet()] state=4 Automated test programs can more easily monitor detailed output, even if the detail is a bit cryptic Automated test programs can more easily monitor detailed output, even if the detail is a bit cryptic These programs cannot easily handle lack of information or ambiguous information. These programs cannot easily handle lack of information or ambiguous information.

20. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Avoid Interactive Testing 20 Interactive tests will rarely be re-done when small changes occurs Interactive tests will rarely be re-done when small changes occurs Spend time designing the tests Spend time designing the tests Create scripts to automatically run tests Create scripts to automatically run tests Emulate input devices using known patterns Emulate input devices using known patterns Ideal Rule: Ideal Rule: Always test the entire system all the time! Always test the entire system all the time! Practical Rule: Practical Rule: Automate testing as much as possible to get close to the ideal rule. Automate testing as much as possible to get close to the ideal rule. Regularly test non-automated parts of the system following documented procedures. Regularly test non-automated parts of the system following documented procedures.

21. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Improper Separation Between ISR and IST 21 ISR: Interrupt Service Routine ISR: Interrupt Service Routine High-priority interrupt, unscheduled High-priority interrupt, unscheduled General rule is to process it and return as soon as possible General rule is to process it and return as soon as possible IST: Interrupt Service Thread IST: Interrupt Service Thread Scheduled actions for most interrupts Scheduled actions for most interrupts Priorities can be managed Priorities can be managed

22. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Bad Implementation of ISR vs IST 22 Following is a very common flaw in many RTOS Driver Implementations Following is a very common flaw in many RTOS Driver Implementations This flaw will usually break the ability to achieve good real-time performance on a system This flaw will usually break the ability to achieve good real-time performance on a system ISR: ISR: disable interrupts disable interrupts decide which IST to run decide which IST to run signal IST signal IST IST: IST: process the interrupt process the interrupt re-enable interrupts re-enable interrupts

23. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Good Implementation of ISR vs IST 23 ISR ISR disable interrupts disable interrupts if quick operation (e.g. < 100 usec) if quick operation (e.g. < 100 usec) perform the full operation perform the full operation else else Mask out the individual interrupt only Mask out the individual interrupt only signal IST signal IST enable interrupts enable interrupts IST IST perform longer operations perform longer operations unmask individual interrupt unmask individual interrupt

24. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Example: Serial Driver 24 ISR ISR Handles individual bytes arriving over serial link Handles individual bytes arriving over serial link It only takes a few microseconds per byte; No need to signal IST for that It only takes a few microseconds per byte; No need to signal IST for that IST IST When a complete packet arrives, or buffer reaches a critical threshold, then IST is signaled to provide more extensive processing When a complete packet arrives, or buffer reaches a critical threshold, then IST is signaled to provide more extensive processing

25. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Why is the good implementation better? 25 Reduces overhead of interrupts that are happening rapidly Reduces overhead of interrupts that are happening rapidly Interrupts may be arriving much faster than the data actually needs to be processed; by processing data less frequently, priority of IST can be lowered, making it easier to schedule Interrupts may be arriving much faster than the data actually needs to be processed; by processing data less frequently, priority of IST can be lowered, making it easier to schedule ISR remains at high priority, thus does not miss any critical events ISR remains at high priority, thus does not miss any critical events Interrupts are never disabled for extended periods of time, thus preventing priority version Interrupts are never disabled for extended periods of time, thus preventing priority version Allows other real-time tasks to execute when needed Allows other real-time tasks to execute when needed

26. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Circular Dependencies 26 This is the root cause of spaghettiaka legacycode!

27. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Circular Dependencies Break Modularity 27 abc defghi jklstu mnouvwpqr xyz Example of Dependency Graph jkl Suppose we want to test or reuse module jkl, or find an error that is observed in jkl.

28. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Circular Dependencies Break Modularity 28 abc defghi jklstu mnouvwpqr xyz Example of Dependency Graph jkl uvwpqr xyz Unit-testing confined to only the four modules shown. If an error is observed is in jkl, then quite likely the issue is in one of the four modules dependent modules. If module jkl needs to be reused in a different system, only the four modules shown are needed.

29. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Circular Dependencies Break Modularity 29 abc defghi jklstu mnouvwpqr xyz Example of Dependency Graph abc defghi jklstu mnouvwpqr xyz jkl uvwpqr xyz abc defghi jklstu mnouvwpqr xyz With just one circular dependency (between jkl and ghi) the number of modules needed to confine errors, unit test, or reuse, is increased.

30. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Circular Dependencies Break Modularity 30 abc defghi jklstu mnouvwpqr xyz abc defghi jklstu mnouvwpqr xyz Example of Dependency Graph abc defghi jklstu mnouvwpqr xyz jkl uvwpqr xyz With just one major circle, jkl cannot be reused or unit tested! abc defghi jklstu mnouvwpqr xyz abc defghi jklstu mnouvwpqr xyz

31. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Circular Dependencies Break Modularity 31 abc defghi jklstu mnouvwpqr xyz abc defghi jklstu mnouvwpqr xyz Such major cycles are quite common: Using a single globals.h type of header that all modules include Low-level device driver directly uses a constant or variable defined by application Software architecture is not layered

32. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Avoiding Circular Dependencies 32 Follow fundamental Software Engineering concepts: Follow fundamental Software Engineering concepts: Data encapsulation and modularity Data encapsulation and modularity Use abstract data types or objects Use abstract data types or objects Avoid including everything Avoid including everything E.g. #include globals.h. This effectively makes everything dependent on everything else. E.g. #include globals.h. This effectively makes everything dependent on everything else. Instead, what you need is what you include Instead, what you need is what you include Anything more causes undesired dependencies Anything more causes undesired dependencies Review component relationships Review component relationships If adding an include file causes a circular dependency, review the design immediately. If adding an include file causes a circular dependency, review the design immediately. This is much easier to fix at the root of the problem, rather than later on down the road. This is much easier to fix at the root of the problem, rather than later on down the road.

33. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com No Code Reviews 33 Textbook Reasons Textbook Reasons Code reviews are a proven way to improve quality and robustness Code reviews are a proven way to improve quality and robustness Studies have shown that more problems can get fixed in one day of code review than in a week of debugging Studies have shown that more problems can get fixed in one day of code review than in a week of debugging These are applicable, but not the main reason this mistake becomes so costly.

34. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Code Reviews 34 Greatest cost for software is in maintenance phase Greatest cost for software is in maintenance phase It is likely that original developers of software are no longer on project, thus not the ones troubleshooting and revising software It is likely that original developers of software are no longer on project, thus not the ones troubleshooting and revising software Code reviews are necessary: Code reviews are necessary: Reviews help eliminate messy code by forcing programmers to show their code to others. Reviews help eliminate messy code by forcing programmers to show their code to others. This results in cleaner code from the start, before the programmer is moved off the project. This results in cleaner code from the start, before the programmer is moved off the project. Reviews double as training sessions to increase number of employees who understand and can maintain the code Reviews double as training sessions to increase number of employees who understand and can maintain the code Reduces company risk that only one employee knows the code, and they decide to leave for any reason. Reduces company risk that only one employee knows the code, and they decide to leave for any reason. Increased consistency for software from one project to the next, and more reuse of code and software architectures Increased consistency for software from one project to the next, and more reuse of code and software architectures

35. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Wrong Priority Assignments 35 Arbitrarily bumping up priorities of important threads is a primary cause of breaking real-time performance and creating priority inversions Arbitrarily bumping up priorities of important threads is a primary cause of breaking real-time performance and creating priority inversions Doing so will help that one thread perform as needed, but at the cost of causing other real-time threads to fail to meet their timing requirements Doing so will help that one thread perform as needed, but at the cost of causing other real-time threads to fail to meet their timing requirements Trying to then modify the priorities of the other threads that stopped working simply counteracts the initial change. Trying to then modify the priorities of the other threads that stopped working simply counteracts the initial change. This can lead to a never-ending debug cycle and a system that never meets requirements. This can lead to a never-ending debug cycle and a system that never meets requirements.

36. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Assigning Priorities 36 Priorities are a system configuration issue Priorities are a system configuration issue Drivers and threads should never define or hard-code their own priorities Drivers and threads should never define or hard-code their own priorities Define all priorities in a tabular configuration file so that priorities of every thread can be seen relative to every other thread. Define all priorities in a tabular configuration file so that priorities of every thread can be seen relative to every other thread. Absolute priorities are meaningless! Absolute priorities are meaningless! Only relative priorities have meaning Only relative priorities have meaning

37. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Assigning Priorities 37 Use Rate Monotonic Algorithm as a Guideline Use Rate Monotonic Algorithm as a Guideline An algorithm for assigning fixed priorities An algorithm for assigning fixed priorities The higher the frequency (or shorter the period) of a task, the higher its priority. The higher the frequency (or shorter the period) of a task, the higher its priority. Semantic importance of a task is irrelevant in real-time scheduling. If a task needs to be hard real-time, it needs to execute faster than other tasks. Semantic importance of a task is irrelevant in real-time scheduling. If a task needs to be hard real-time, it needs to execute faster than other tasks. While an accurate analysis would be nice, it is not necessary unless on the threshold of performance While an accurate analysis would be nice, it is not necessary unless on the threshold of performance Consider the general rates and approximate utilization of each driver or thread, and assign priorities accordingly Consider the general rates and approximate utilization of each driver or thread, and assign priorities accordingly

38. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Prioritizing Random Events – Right Way 38 The Right Way is to define Aperiodic Servers The Right Way is to define Aperiodic Servers Each server has reserved execution time at high priority, and has the ability to use leftover execution time at lower priorities Each server has reserved execution time at high priority, and has the ability to use leftover execution time at lower priorities Requires appropriate RTOS mechanisms to detect timing errors Requires appropriate RTOS mechanisms to detect timing errors E.g. missed deadline or using more CPU time than reserved. E.g. missed deadline or using more CPU time than reserved. Unfortunately, many RTOS dont support the mechanisms Unfortunately, many RTOS dont support the mechanisms See Mechanisms for Detecting and Handling Timing Errors by D. Stewart and P. Khosla See Mechanisms for Detecting and Handling Timing Errors by D. Stewart and P. Khosla Available on the Web; search by title Available on the Web; search by title Use the Right Way for Hard Real-Time Systems Use the Right Way for Hard Real-Time Systems

39. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Prioritizing Random Events – Quick Way 39 Estimate the fastest the random event arrives, consider that the frequency of the thread Estimate the fastest the random event arrives, consider that the frequency of the thread Estimate how long each event takes, and use that as utilization of the thread Estimate how long each event takes, and use that as utilization of the thread Assign priority using Rate Monotonic Algorithm as though this was periodic Assign priority using Rate Monotonic Algorithm as though this was periodic This method is generally good enough for soft-real-time threads This method is generally good enough for soft-real-time threads It can be used if there are hard-real-time threads It can be used if there are hard-real-time threads as long as the highest priority assigned to a random event thread is less than any hard real-time thread as long as the highest priority assigned to a random event thread is less than any hard real-time thread

40. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Too Much Locking and Blocking 40 Locking and Blocking comes in many forms Locking and Blocking comes in many forms Semaphores Semaphores Message Passing Message Passing Mutex Mutex Critical Sections Critical Sections Blocking is an enemy to real-time predictability! Blocking is an enemy to real-time predictability! Reduces effectiveness of real-time scheduling Reduces effectiveness of real-time scheduling Introduces priority inversion Introduces priority inversion Potential exists for deadlock if multiple locks Potential exists for deadlock if multiple locks

41. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Eliminate Locking when Possible 41 Minimize inter-module communication Minimize inter-module communication Too much inter-module communication is a sign of a poorly-decomposed software architecture Too much inter-module communication is a sign of a poorly-decomposed software architecture Limit real-time scheduling preemption Limit real-time scheduling preemption Systems dont have to be fully preemptive Systems dont have to be fully preemptive If module A reads data that module B runs, no reason one should have to interrupt the other; they can be set to same priority, and always execute in sequence If module A reads data that module B runs, no reason one should have to interrupt the other; they can be set to same priority, and always execute in sequence

42. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Eliminate Locking when Possible 42 If using message passing: If using message passing: Use non-blocking mechanisms only, such as asynchronous messages Use non-blocking mechanisms only, such as asynchronous messages If no message available, then continue with alternate execution If no message available, then continue with alternate execution Use large enough queues Use large enough queues In soft-real-time systems, if they fill up, throw away data In soft-real-time systems, if they fill up, throw away data In hard real-time systems, CPU utilization must be low enough on average so that queues never fill up In hard real-time systems, CPU utilization must be low enough on average so that queues never fill up

43. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Eliminate Locking when Possible 43 Critical Sections Critical Sections Sometimes it is ok to disable specific interrupts for a few microseconds, to eliminate a lock and possible blocking Sometimes it is ok to disable specific interrupts for a few microseconds, to eliminate a lock and possible blocking Introduce a buffer for data shared between interrupts and threads Introduce a buffer for data shared between interrupts and threads interrupt only accesses buffer interrupt only accesses buffer thread manages refreshing real data from the buffer. thread manages refreshing real data from the buffer. Access buffers and FIFOs using algorithms that guarantee integrity of data without the need for locks Access buffers and FIFOs using algorithms that guarantee integrity of data without the need for locks Easiest to achieve if only single producer and single consumer Easiest to achieve if only single producer and single consumer Guard hardware registers by using servers for individual resources Guard hardware registers by using servers for individual resources Can then use asynchronous message passing between applications (clients) and the server Can then use asynchronous message passing between applications (clients) and the server

44. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Block threads ONCE and ONLY ONCE 44 Ideal model of a real-time thread is to block once per cycle, waiting for start of a cycle Ideal model of a real-time thread is to block once per cycle, waiting for start of a cycle Once cycle begins, never block within the cycle Once cycle begins, never block within the cycle Read Inputs/Events Do Processing Write Outputs Wait for Event (BLOCK) For periodic tasks, event is time-based For other tasks, event could be an interrupt, message arrival, semaphore wakeup, or any other signal

45. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com No measurements of execution time! 45

46. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com For Real-Time Systems: Execution Time = $$$ 46 Money ($$$) Money ($$$) Have a limited amount Have a limited amount Bills have deadlines Bills have deadlines Cant spend more than you have* Cant spend more than you have* Count it to know how much you are spending Count it to know how much you are spending Dont buy without knowing the price Dont buy without knowing the price Good budgeting leads to good cash flow Good budgeting leads to good cash flow CPU Execution Time CPU Execution Time Have a limited amount Have a limited amount Real-time deadlines Real-time deadlines Cant use more than you have* Cant use more than you have* Measure it to know how much you are using Measure it to know how much you are using Dont execute code without knowing its CPU usage Dont execute code without knowing its CPU usage Good real-time design leads to good CPU usage Good real-time design leads to good CPU usage * unless you use credit * unless you use credit

47. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Measuring Execution Time 47 Design your system so that the code is measurable! Design your system so that the code is measurable! Measure execution time as part of your standard testing. Measure execution time as part of your standard testing. Do not only test the functionality of the code! Do not only test the functionality of the code!

48. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Measuring Execution Time 48 Learn both coarse-grain and fine-grain techniques to measure execution time Learn both coarse-grain and fine-grain techniques to measure execution time Use coarse-grain measurements for analyzing real-time properties Use coarse-grain measurements for analyzing real-time properties Use fine-grain measurements for optimizing and fine-tuning Use fine-grain measurements for optimizing and fine-tuning See Measuring Execution Time and Real-Time Performance by D. Stewart See Measuring Execution Time and Real-Time Performance by D. Stewart Available on web by searching by title Available on web by searching by title

49. DesignWest 2012 – San Jose Costly Mistakes with Real-Time Software Development Dave Stewart, PhD © 2012 InHand Electronics, Inc. – www.inhand.com Costly Mistakes of Real-Time Software Development 49 #1No Measurements of Execution Time! #2Too Much Locking and Blocking #3Wrong Priority Assignments #4No Code Reviews #5Circular Dependencies #6Improper Separation between ISR and IST #7Inadequate Design for Test #8Poor or No Software Design Diagrams #9 Its Just a Glitch #10No Emulators of Target Application