1. UCDavis, ecs150 Spring 2006 04/21/2006ecs150, spring 20061 Operating System ecs150 Spring 2006 : Operating System #3: Priority Inversion (paper) Dr. S. Felix Wu Computer Science Department University of California, Davis http://www.cs.ucdavis.edu/~wu/ sfelixwu@gmail.com

2. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 20062 Priority Scheduling l A priority number (integer) is associated with each process l The CPU is allocated to the process with the highest priority (smallest integer  highest priority). –Preemptive –Non-preemptive l SJF is a priority scheduling scheme where priority is the predicted next CPU burst time. l FCFS is a priority scheduling scheme where priority is the arrival time. l Lottery scheduling is a probabilistic priority scheduling scheme where the priority is the ticket number.

3. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 20063 “Fixed” Priority l What is it? –The process sticks with the origin assigned priority. l A good or bad idea? –Have we learned the lesson from HW#2? l What other possible policy? –Dynamic policy. l Problem  Starvation – low priority processes may never execute. l Solution  Aging – as time progresses increase the priority of the process.

4. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 20064 1 0 0 1 0 1 ::.::. 256 different priorities  64 scheduling classes RR

5. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 20065 Real-Time Scheduling l Hard real-time systems – required to complete a critical task within a guaranteed amount of time. –Resource reservation- guarantees on time completion or rejects process l Soft real-time computing – requires that critical processes receive priority over less fortunate ones.

6. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 20066 Unexpected Effects between two OS control mechanisms l Real-time priority scheduling –Responsiveness: if a higher priority thread appears, serve it asap. l Mutual exclusion –Integrity: if a higher priority thread wants to enter a critical section being hold by a lower priority thread, it has to wait for the lower priority thread to leave “the critical section”.

7. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 20067 pthread_mutex_lock pthread_mutex_unlock 1 0 0 1 0 1 ::.::. 256 different priorities  64 scheduling classes RR

8. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 20068 Real-Time Threads l Thread τ 1 L L L R x L l Thread τ 2 L L... L l Thread τ 3 L L L R x L... L l L: local CPU burst l R: resource required (Mutual Exclusion)

9. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 20069 Example l Suppose that threads τ 1 and τ 3 share some data. l Access to the data is restricted using semaphore x: –each task executes the following code: l do local work (L) l sem_wait(s) (P(x)) –access shared resource (R) l sem_signal(s) (V(x)) l do more local work (L)

10. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200610Blocking τ 2 τ 3 t0t+3 t+4 R L L L R R L τ 1 t+6 L L L Blocked!

11. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200611 The middle thread τ 2 τ 3 t0t+3 L L L R τ 1 L L L Blocked! t+2

12. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200612 Unbounded Priority Inversion τ 2 τ 3 t0t+3 t+253 RL L L R R L τ 1 t+254 L L L... L Blocked! t+2

13. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200613 Unbounded Priority Inversion τ 2-1 τ 3 t0t+3 t+2530 RL L L R R L τ 1 t+2 t+2540 L L L L Blocked! τ 2-2 τ 2-n L L

14. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200614 The problem.. l Do we have Priority Inversion in the FreeBSD kernel with XYZ scheduling policy? l As long as we have priority and mutual exclusion at the same time, we will have some form of priority inversion. l How to resolve it? trade-off?

15. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200615 Priority Inheritance τ 2 τ 3 t0t+3t+4 L... L L L L R R L τ 1 t+2t+6 L L L R Blocked! dynamic  3 =  1 L... L

16. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200616 Priority Inheritance Protocols l L. Sha, R. Rajkumar, J. Lehoczky, “Priority Inheritance Protocols: An Approach to Real-Time Synchronization”, IEEE Transactions on Computers, Vol. 39, No. 9, pp. 1175-1185, 1990

17. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200617 The MARS Pathfinder Problem l “But a few days into the mission, not long after Pathfinder started gathering meteorological data, the spacecraft began experiencing total system resets, each resulting in losses of data. The press reported these failures in terms such as "software glitches" and "the computer was trying to do too many things at once".” …

18. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200618 The MARS Pathfinder Problem l “VxWorks provides preemptive priority scheduling of threads. Tasks on the Pathfinder spacecraft were executed as threads with priorities that were assigned in the usual manner reflecting the relative urgency of these tasks.” l “Pathfinder contained an "information bus", which you can think of as a shared memory area used for passing information between different components of the spacecraft. A bus management task ran frequently with high priority to move certain kinds of data in and out of the information bus. Access to the bus was synchronized with mutual exclusion locks (mutexes).”

19. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200619 “The meteorological data gathering task ran as an infrequent, low priority thread, and used the information bus to publish its data. When publishing its data, it would acquire a mutex, do writes to the bus, and release the mutex. If an interrupt caused the information bus thread to be scheduled while this mutex was held, and if the information bus thread then attempted to acquire this same mutex in order to retrieve published data, this would cause it to block on the mutex, waiting until the meteorological thread released the mutex before it could continue. The spacecraft also contained a communications task that ran with medium priority.” High priority: retrieval of data from shared memory Medium priority: communications task Low priority: thread collecting meteorological data

20. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200620 l “Most of the time this combination worked fine. However, very infrequently it was possible for an interrupt to occur that caused the (medium priority) communications task to be scheduled during the short interval while the (high priority) information bus thread was blocked waiting for the (low priority) meteorological data thread. In this case, the long-running communications task, having higher priority than the meteorological task, would prevent it from running, consequently preventing the blocked information bus task from running. After some time had passed, a watchdog timer would go off, notice that the data bus task had not been executed for some time, conclude that something had gone drastically wrong, and initiate a total system reset. This scenario is a classic case of priority inversion.”

21. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200621 l Priority inheritance also solved the Mars Pathfinder problem: the VxWorks operating system used in the pathfinder implements a flag for the calls to mutex primitives. This flag allows priority inheritance to be set to “on”. When the software was shipped, it was set to “off”. The problem on Mars was corrected by using the debugging facilities of VxWorks to change the flag to “on”, while the Pathfinder was already on the Mars [Jones, 1997].

22. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200622 Basic Priority Inheritance l For each resource (semaphore), a list of blocked threads must be stored in a priority queue. l A thread τ i uses its assigned priority, unless it is in its critical section and blocks some higher priority threads, in which case, thread τ i uses ( inherits ) the highest dynamic priority of all the threads it blocks. l Priority inheritance is transitive; that is, if thread τ i blocks τ j and τ j blocks τ k, then τ i can inherit the priority of τ k.

23. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200623 mutex priority inheritance pthread_mutex_lock pthread_mutex_unlock ttt waiting queue t priority

24. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200624 pthread_mutex_lock pthread_mutex_unlock M1M1 ttt waiting queue t priority pthread_mutex_lock pthread_mutex_unlock M2M2 ttt waiting queue priority

25. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200625 Transitive Priority pthread_mutex_lock pthread_mutex_unlock M1M1 ttt waiting queue t priority pthread_mutex_lock pthread_mutex_unlock M2M2 tt waiting queue t priority

26. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200626 Types of Blocking l Direct - thread τ 1 and τ 2 use a shared resource. If the low priority thread is in its critical section, then it directly blocks the high priority thread. l Indirect (push-through) - if a low priority thread inherits the priority of a high priority thread, a medium priority thread can be blocked while the low priority thread is in its critical section.

27. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200627 Properties of Priority Inheritance l Under the basic priority inheritance protocol, if there are m semaphores that can block a thread J, then J can be blocked at most m times; i.e., on each semaphore at most once.

28. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200628 Any Problems with the basic Priority Inheritance Protocol? l ???

29. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200629Problems l The Basic Priority Inheritance Protocol has two problems: –Deadlock - two threads need to access a pair of shared resources simultaneously. If the resources, say A and B, are accessed in opposite orders by each thread, then deadlock may occur. –Blocking Chain - the blocking duration is bounded (by at most the sum of critical section times), but that may be substantial.

30. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200630 pthread_mutex_lock pthread_mutex_unlock M1M1 ttt waiting queue t priority pthread_mutex_lock pthread_mutex_unlock M2M2 ttt waiting queue priority t

31. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200631 Blocking Chain Example l Task  1 : L R 2 L R 3 L R 4 L... L R n L,2(n-1) l Task  2 : L R 2 R 2, 2(n-2) l Task  3 : L R 3 R 3, 2(n-3) l Task  4 : L R 4 R 4, 2(n-4) l... l Task  n-1 : L R n-1 R n-1, 2(n-(n-1)) l Task  n : L R n R n, 2(n-n) starting time

32. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200632 Blocking Chain τ 2 τ n 0 L R n τ 1 RnRn R n L L R 2 R2R2 Blocked! L R 2 L Blocked!

33. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200633 Different Timing? τ 2 τ n 0 L R n τ 1 RnRn R n L L R 2 R2R2 Blocked! L R 2 L Blocked! ???

34. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200634 Different Timing?? l Task  1 : L R 2 L R 3 L R 4 L... L R n L,2(n-1) l Task  2 : L R 2 R 2, 2(n-n) l Task  3 : L R 3 R 3, 2(n-(n-1)) l Task  4 : L R 4 R 4, 2(n-(n-2)) l... l Task  n-1 : L R n-1 R n-1, 2(n-3) l Task  n : L R n R n, 2(n-2) starting time How many times Task  1 will be blocked?

35. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200635 Priority Ceiling Protocols (PCP) l A higher priority thread can be blocked at most once, in its life time, by one lower priority thread. l Deadlocks are prevented/avoided (?!). l Transitive inheritance is prevented. l Are they really critical?

36. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200636 PCP l How do we accomplish these goals intuitively?

37. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200637 Locking a Mutex l If the “mutex M” is available and “thread T” needs it, should T lock it? pthread_mutex_lock pthread_mutex_unlock t Mutex?? Mutex + Priority Inheritance??

38. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200638 Risk for Locking a Mutex l If the “mutex M” is available and “thread T” needs it, should T lock it? pthread_mutex_lock pthread_mutex_unlock t t Checking before Locking it!! We don’t know whether the high priority thread will occur in the next X seconds! But, does it matter?

39. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200639 “Checking” What??

40. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200640 “Checking” What?? l What is our goal? –High priority thread will be blocked at most once. –We will allow blocking ONCE. l Idea of the check: –If we are the first Mutex, we lock it. –If we are not the first, we will not. l But, how to design/implement this idea?

41. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200641 PCP l The protocol uses the notion of a system-wide mutex ceiling priority. l Each thread has a static default priority assigned. l Each resource (mutex) has a static ceiling priority defined to be the maximum static priority of any thread that uses it. l Each thread has a dynamic priority equal to the maximum of its own default priority and any priority it inherits due to blocking a higher priority thread.

42. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200642 mutex priority ceiling pthread_mutex_lock pthread_mutex_unlock ttt potential customers PC priority A preventive action (could be unnecessary though)

43. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200643PCP l At run-time, if a thread wants to lock a mutex, its priority must be strictly higher than the ceilings of all mutexes currently locked by other threads (unless it is the thread holding the lock on the mutex with the highest ceiling). l If this condition is not satisfied, then the thread is blocked. l When a thread is blocked on a mutex, the thread currently holding the waited mutex inherits the priority of the blocked thread.

44. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200644 Priority Ceiling Should I get it? PC locked unlocked MaxPC value thread t 2 ??? 4 thread t 4 2 thread t 3 2 4

45. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200645 Mutex/PIP Get it as long as it is available! PC locked unlocked MaxPC value thread t 2 YES 4 thread t 4 1 thread t 3 2 4

46. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200646 PCP Not so Fast PC locked unlocked MaxPC value thread t 2 NO 4 thread t 4 1 thread t 3 2 4

47. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200647 PCP Not so Fast PC locked unlocked MaxPC value thread t 2 NO 4 thread t 4 2 thread t 3 3 4

48. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200648 PCP How about??? PC locked unlocked MaxPC value thread t 2 ?? 4 thread t 4 2 thread t 2 3 4

49. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200649 PCP mutex blocking PC locked unlocked MaxPC value thread t 1 ?? thread t 2 thread t 3

50. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200650 Are we sure about the claim of PCP? l A higher priority thread can be blocked at most once, in its life time, by one lower priority thread. l Deadlocks are prevented/avoided. l Try to find a “Counter Example” to show that PCP’s claim is FALSE!!

51. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200651 4 3 2 1 024681012141618 Executing Executing with Q locked Preempted Executing with V locked Blocked Tasks Ceiling-driven Indirectly Blocked

52. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200652 4 3 2 1 024681012141618 Executing Executing with Q locked Preempted Executing with V locked Blocked Priority Inversion Ceiling-driven Indirectly Blocked

53. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200653 4 3 2 1 024681012141618 Executing Executing with Q locked Preempted Executing with V locked Blocked Priority Inversion Area Ceiling-driven Indirectly Blocked

54. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200654 111 4 3 2 1 024681012141618 Process 1 Basic Priority Inheritance

55. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200655 2212 4 3 2 1 024681012141618 Process 1 Priority Ceiling 11 locked

56. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200656 2222 4 3 2 1 024681012141618 Process 1 Priority Ceiling (1 delay) 11 locked

57. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200657 Can we do better??

58. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200658 1111 a b c d 024681012141618 process 11 Priority Ceiling Emulation 11

59. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200659 Priority Ceiling Emulation l Each thread has a static (base) default priority assigned (perhaps by the deadline monotonic scheme). l Each resource has a static ceiling value defined, this is the maximum priority of the threads that use it. l A thread has a dynamic (active) priority that is the maximum of its own static priority and the ceiling values of any resources it has locked l As a consequence, a thread will only suffer a block at the very beginning of its execution l Once the thread starts actually executing, all the resources it needs must be free; if they were not, then some thread would have an equal or higher priority and the thread’s execution would be postponed

60. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200660 Semaphore Requirements l Threads must lock and unlock semaphores in a “nested” or “pyramid” fashion: –Let P(S) = L(S) = lock(S) = sem_wait(S). –Let V(S) = U(S) = unlock(S) = sem_signal(S). –Example: P(s 1 );P(s 2 );P(s 3 );...;V(s 3 );V(s 2 );V(s 1 ); s 1 s 2 s 3

61. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200661 J, Pri(J) = 1I, Pri(I) = 120 J,80%IJI IJJJIJJJJJJJIJJJJIJJ

62. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200662 Lottery Ticket Sharing l If task I is blocked by task J, all I’s tickets will be used by J until the block is gone.

63. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200663 LT Sharing vs. PI l LTS: summation of tickets for ALL waiting processes. –With Lottery Tickets, we can ADD “priority quantities” together. l PI: the highest priority among ALL waiting processes. –What is the semantic meaning of “adding” priorities together?

64. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200664 4 3 2 1 024681012141618 priority time 4 3 2 1 024681012141618 priority time

65. UCDavis, ecs150 Winter 2006 04/21/2006ecs150, spring 200665 4 3 2 1 024681012141618 priority time eeVQeF eeQQeF eeeF eVVVeF