1. Achieving (hard) Real-Time with General Purpose Operating Systems

2. Motivation There is a conflict between the requirements of real-time systems and the need to run common day applications. Can we use a general purpose OS for real-time applications?

3. Outline Basic definitions The challenges Can we use existing systems Proposed solutions Case studies of real-time oss.

4. Basic definitions A task is the basic unit of execution. Each task has three important properties: 1. release time: the point in time from which the task can be executed. 2. deadline: the point in time by which the task must complete. 3. execution time: the time the task takes to execute E.g. brakes controller

5. example If we have the following set of tasks: task1: release=0 deadline=3 exec time=2 task2: release=2 deadline=5 exec time=1 task3: release=6 deadline=10 exec time=3 Then they must be scheduled during these periods: 23456789101110

6. Basic definitions A processor is a concept used to denote a resource that is active in the execution of a task –Usually refers to the CPU.

7. Classification of task timing constraints There are two types of timing constraints hard and soft. Definitions: if missing a deadline is a fatal error or has disastrous consequences it is considered hard (missing an occasional deadline is not acceptable) otherwise (if some percentage of missed deadlines is acceptable) it is soft. Classic examples: braking systems controller => hard multimedia streaming => soft

8. Soft real time systems Contain only tasks with no hard timing constraints. Also known as “best effort” systems Most modern operating systems can serve as the base for a soft real time systems. Examples: multimedia transmission and reception, networking, telecom (cellular) networks, web sites and services, computer games.

9. Hard real time systems Contains tasks with hard timing constraints. Requires formal verification/guarantees of being to always meet its hard deadlines (except for fatal errors). Examples: air traffic control, vehicle subsystems control, medical systems

10. Classifications of real time systems The type of system can be either a uni- processor, mp or distributed system. Moving to an mp or distributed real time systems adds lots of difficulty by requiring real time communication and synchronization mechanisms between the processors.

11. Classifications of real time systems There are two different execution models: In a preemptive model of execution a task may be interrupted (preempted) during its execution and another task run in its place. In a non-preemptive model of execution after a task that starts executing no other task may execute until this task concludes or yields the CPU.

12. Classifications of real time systems The task model for a real time system has two main types: 1. purely periodic: every task is cyclic and executes periodically, even IO is polled. The nature of tasks doesn’t vary much between cycles. 2. aperiodic/asynchronous: most tasks in the system aren’t periodic.

13. Classifications of real time systems A static system is a system where all tasks are knows at design time including their release times (full apriory knowledge). A dynamic systems is a system that can dynamically create and destroy tasks at runtime (No full apriory knowledge).

14. Classifications of real time systems In many real time systems the tasks have different priorities. There are two possible models : static-priorities : the priorities of tasks don’t change during execution dynamic-priorities the priority of tasks may change during execution

15. Classifications of real time systems Sets of tasks are classified according to the way they share resources: We will call a task set dependent if the tasks in it share resources. We will call a task set independent if they do not.

16. Outline Basic definitions The challenges Can we use existing systems Proposed solutions Case studies of real-time oss.

17. The main challenges Processor sharing – scheduling Resource sharing Handling external input / communication

18. Scheduling a scheduler needs to decide when to run each task in order to meet the deadlines imposed on the tasks. In order to do this the scheduler needs to base its operation upon the properties of the tasks (release times, deadlines,etc..).

19. Scheduling If scheduling is done online, the overhead of scheduling needs to be reasonable (this can be hard because most of the optimal scheduling algorithms are np-hard, even for a single processor). If the task model is dynamic we need the scheduler to decide if adding some new task would prevent it from filling obligations to currently running tasks (admissibility testing).

20. Resource sharing When tasks share resources they become dependant on one another and this limits our flexibility when scheduling, requiring us to use synchronization mechanisms to protect critical sections in the tasks. This raises the issues of deadlocks, process starvation and priority inversion.

21. Handling external input / communication In real time system there will often be constraints on the minimal time until the system processes input/communication. These constraints must be reflected in the scheduling of tasks, the handling of external interrupts and the communication infrastructure and protocols chosen.

22. Outline Basic definitions The challenges Deficiencies in modern operating systems Proposed solutions Case studies of real-time oss.

23. Why most modern OSs are unsuitable The basic guiding concept for most operating systems and schedulers is to strive for good average performance while in real time systems meeting the deadlines is far more important than average performance. many modern operating features are problematic in a hard real time setting.

24. Problematic modern features Protected memory: while important for the stability of the system it carries a high overhead in a real time environment because of the added cost to a context switch and the caches that need to be flushed each context switch ( the tlb for example). User and kernel modes: while protecting the systems from user processes adds significant overhead because each mode switch incurs a performance penalty because of the trap instructions and cache flushes. memory paging: can increase the cost of a context switch by several magnitudes if the memory pages we need were switched out. This causes a lot of unpredictability in the timing.

25. Problematic modern features Dynamic memory allocation: this is a critical feature used by most modern application, however it creates memory fragmentation and as a result adds to timing unpredictability.

26. Linux Linux Is based on UNIX which is a time sharing system designed to optimize average performance (as weighted by priorities) and Linux preserves this nature. This means that as the load increases the real time processes will suffer the most. Linux currently doesn’t support an interface for telling the OS that a task is real time or informing it of any timing constraints on a task.

27. Linux The basic Linux scheduler is a time slice based on priorities, however it is a “fair” scheduler and favors interactive (IO bound) programs rewarding programs who haven’t used up all of their CPU share. This presents several problems: 1. the time slice resolution may be higher than requires for the timing and so it may be impossible to meet the timing demands. 2. raising the priority of real time tasks is insufficient because enough lower priority tasks will cause us to miss deadlines anyway. 3.fairness is not a desirable quality in real-time systems.

28. Linux Linux does not provide a reliable mechanism to wake a task up at a certain time. The sleep timer can only promise to wake a task up after a certain time. The kernel can’t be preempted so long system calls executed in the kernel can interfere with timing constraints. External interrupts are disabled at several sections in the kernel to protect OS data from corruption (saves the effort of making them reentrant). This adds unpredictability to the amount of time it takes to respond to IO.

29. Linux – kernel 2.6 changes These has been some progress on some of these issues in the 2.6 Linux kernel: 1.the kernel is now preemptable although not all of it is. 2.improvements in IO handling. despite these changes 2.6 is still not suitable for hard real time performance.

30. Windows Windows (NT based) suffers from most of the same problems Linux does. Windows doesn’t support the specification of timing constraints for tasks.

31. Windows Interrupts may be delayed for an arbitrarily long time in critical sections of the kernel. Handling of interrupts is done in two parts the second part the deferred procedure call is executed later in a FIFO manner and this may cause delays if a lot of DPCs are awaiting dispatch. Requests for synchronization objects such as semaphores are also done in FIFO, leading to similar unpredictable delays.

32. Windows The windows scheduler supports threads on an OS level. Each thread is given a priority (0-31). The scheduler has compensation for input bound threads and anti-starvation mechanisms. Windows has a “real time” priority for threads but other real-time threads and the kernel can preempt a task with real time priority (15-26). A “real time” priority is higher than some parts of the OS, causing most non-real-time apps to become unusable

33. Fixing the problems Fixing these problems in either OS would require either some form of “hack” into the operating systems design or a massive rewrite of the kernel. This has been tried using both of the operating systems as we will see in later case studies (rt-linux, rialto).

34. Outline Basic definitions The challenges Deficiencies in modern operating systems Proposed solutions Case studies of real-time OSs.

35. Scheduling Scheduling algorithm properties: 1.An algorithm is said to be optimal if for each task set for which a feasible scheduling is possible the algorithm will find one (assumes tasks are independent). 2.Schedulable utilization is the maximal utilization of a task set such that a task load under this utilization will be scheduled successfully by the algorithm.

36. Cyclic executive This is a static table driven approach to scheduling. It assumes full apriori knowledge meaning tasks are static and periodic. The schedule is created offline, so scheduling doesn’t load the system during runtime. performance guarantees can be given before system is built. The “algorithm”: analyze and schedule tasks offline using some algorithm which creates a static schedule for running all tasks during runtime.

37. Cyclic executive The cyclic executive approach can incorporate tasks of which there is no foreknowledge by scheduling them in empty spaces in the cycle, however it can’t make any guarantees about their timing. This approach works for the other algorithms as well.

38. Priority driven algorithms This is a family of algorithms which gives the tasks priorities according to some criterion and schedules at each point the task with the highest priority. They are efficient in that if there are tasks ready to execute the CPU is always busy. This approach is attractive because in case of performance degradation the most important tasks will be affected the least.

39. Rate monotonic This algorithm assumes a periodic task model with preemption. The Algorithm: is a priority driven algorithm where the tasks are assigned priorities proportional to their cycle length, the shorter the cycle the higher the priority.

40. Rate monotonic The algorithm is optimal in a way: if there is a feasible schedule for a periodic task load where P i <=D i on a single processor with fixed priorities then RM will find a feasible schedule. Liu and Leyland show that the schedulable utilization of RM is bounded by : where N is the number of real time tasks

41. example If we have the following set of tasks: task1: period=deadline=3 execution time=1 task2: period=deadline=5 execution time=3 Then the schedule will look like this: 234567891010 1 1 2 1 3 4 22

42. The problem with static priority driven algorithms This simple example shows that a schedulable set of tasks can’t be scheduled based on static priorities 234567891010 1 1 2 3 1 task1: period=deadline=4 execution time=2 task2: period=deadline=10 execution time=5 Task 1 must have higher priority Task 2 must have higher priority

43. EDF algorithm The Earliest Deadline First algorithm is a priority driven algorithm in which the tasks get priorities according to how close their nearest deadline is. The closer the deadline the higher the priority.

44. example If we have the following set of tasks: task1: period=deadline=3 execution time=1 task2: period=deadline=5 execution time= 3 Then the schedule will look like this: 234567891010 1 1 2 3 4 2 2

45. EDF properties The algorithm is optimal. The schedulable utilization of the algorithm is 1. Priorities must only be recalculated when a new task arrives, as time advances equally for all tasks. the algorithm behaves unpredictably at overloads, and suffers more from overloads than RM.

46. LST algorithm The slack time of a task is the amount of time until the task must be scheduled or miss it’s deadline. The least slack time algorithm is a priority driven algorithm in which the tasks receive priorities according the their slack times. The shorter the slack time the higher the priority. The LST algorithm is optimal

47. Priority inversion The priority inversion problem exists in the preemptive model when tasks are dependant on one another (share resources). it causes a high priority task to wait for a low priority task to conclude because it is holding a critical resource. This problem can be at least partially solved by priority inheritance, or the priority ceiling protocol based on a similar concept.

48. Outline Basic definitions The challenges Deficiencies in modern operating systems Proposed solutions Case studies of real-time OSs

49. Case study We will go over how some actual implementations of real time operating systems tackled these problems. Rt-Linux Rialto VxWorks

50. Rt-linux RT-Linux is a system developed as a way to adapt Linux to hard real time performance and still give the user most of the advantages of the Linux environment. The main idea: add a hard real time scheduler into (under) the kernel which will schedule the real time tasks and will run the rest of Linux as the lowest priority task.

51. Rt-linux the main philosophy is this: Real time programs need to be split into parts, a small light parts with hard real time constraints and larger parts that do most of the processing. the light real time parts will be scheduled in the real time scheduler, while the heavier parts will run as normal processes under Linux. The two parts can communicate through a real- time-FIFO, a non-blocking queue which allows the real time parts to read and write data from normal processes.

52. Rt-linux Real time tasks are implemented as kernel modules whose init function sends the real time scheduler all the information about their release times, periods and deadlines. Rt-Linux allows for swappable schedulers for the real time tasks, a scheduler based on the EDF algorithm and another based on the RM algorithm. Linux is run as the lowest priority process in either case and will only execute if the real time scheduler has nothing else to do. When Linux is running it uses it’s own scheduler to schedule tasks running under it.

53. Rt-Linux The Linux kernel runs as a process under the real time scheduler and as a result can be preempted by it at any time, and while this allows for good sleep timers and scheduling flexibility it dictates that system calls can’t be used by real time tasks in order to preserve the consistency of the kernels internal state.

54. Rt-Linux All the memory needs of the real time processes are handled by the real time subsystem (no system calls) and memory is allocated in non- pageable blocks saving the uncertainty of swap file usage. As all the real time tasks run in the same kernel memory space we lose the benefits of protected memory for them but context switches are much faster because we save the cost of flushing the TLB cache and using trap instructions to enter and exit kernel mode for scheduling.

55. Rt-Linux A very serious problem is the disabling of interrupts by the kernel this is handled by replacing the function call to disable interrupts in the kernel source with a macro that simply masks these interrupts from the Linux process but not from the other real time processes, so interrupts are never blocked as far as the real time scheduler or tasks are concerned. The masking of the interrupts is done by having the real time portions of the system initially handle all interrupts and pass the ones intended to the Linux process on to it as software interrupts (in case Linux thinks the interrupts are enabled).

56. Rt-Linux - limitations Doesn’t handle the problem of scheduling dependant real time tasks. Not all applications can be split into a light real-time parts and non-real-time parts. Not suited for dynamic real time systems.

57. Rt-Linux - summary Demonstrates the “hack in to the architecture” approach to adding real time. Is a functional system used in the wild. It’s requirements form the real time portions of the tasks may make it unsuitable for some applications.

58. Rialto Rialto is a real time system developed at Microsoft Research based on the windows NT architecture. The main idea: rewriting the kernel and scheduler so that they support real time deadlines.

59. Rialto Unlike rt-linux it doesn’t require the separation of the hard real time functionality from the rest of the code. It uses the beginConstraint and endConstriant function calls that pass the parameters of the real time portion of task to the scheduler.

60. Rialto The parameters passed to the scheduler: - Release time - Deadline - Running time estimate - Criticality (critical=hard constraint, non-critical=soft ) The scheduler returns the feasibility of scheduling the task. This allows the task to perform load shedding if possible, or fail gracefully if not.

61. Rialto the scheduling in Rialto is quite complicated: 1.Tasks can be granted CPU time reservations. This is the average amount of CPU time they should get. 2.The scheduler is based on a variation of preemptive LST which uses the maximal amount of time before a task has to start running (deadline-running time estimate). 3.Critical tasks always run before non-critical tasks.

62. Rialto The rialto system is system-resource aware. Tasks can negotiate with the resource manager in order to reserve the resources they need. The resource manager arbitrates the use of resources, and can cause an existing task to relinquish resources allocated to it.

63. Rialto - limitations Rialto doesn’t handle the problem of the kernel disabling interrupts. The schedulability test was not fully implemented originally.

64. Rialto -summary Rialto uses the approach of rewriting an existing kernel in order to allow support for real time performance. Built as “academic” work,was never widely used. Uses a novel scheduling approach, But because it is complicated it is sensitive to many implementation details.

65. VxWorks VxWorks is a commercial hard real time operating system developed by wind river systems. The main idea: use monolithic kernel to schedule user tasks according to user defined priorities. Maximize kernel timing predictability. Gives the users maximal control.

66. VxWorks A dedicated real time system, not intended as a general purpose OS. lacks many modern os features that interfere with real time performance (flat memory model, no paging).

67. VxWorks Scheduling is done using a preemptive priority driven approach, priorities are chosen arbitrarily by the user (0-256). Priorities can be changed by the user at runtime but this is discouraged. A user can lock a task so that it can’t be preempted even by higher priority tasks or interrupts. This allows the use of the fixed priority response time analysis to check schedulability offline.

68. VxWorks Is resource sharing aware and has a priority inheritance built in. Optimizations in implementation of the context switch and the return from interrupts. The kernel never disables NMI (non- maskable interrupts) so they are always available to the user.

69. VxWorks - limitations Lacks many modern OS features. Guaranteeing the deadlines is the responsibility of the user at design time. Doesn’t support most modern applications and APIs (only a small subset of POSIX). Despite the flat memory model dynamic memory allocation still cases memory fragmenting, which increases timing unpredictability.

70. VxWorks - summary A dedicated and widely used real time system. Offers the user maximal control, but also passes him responsibility for the deadlines. Lacks many modern OS features.

71. Summary This talk reviewed: - The basic concepts of real time systems - The challenges of building a real time system - The use of generic OSs as real time systems - Some Solutions to the Challenges. - The approaches taken by some real systems

72. questions