1. Introduction Companion slides for The Art of Multiprocessor Programming by Maurice Herlihy & Nir Shavit

2. Art of Multiprocessor Programming2 Moore’s Law Clock speed flattening sharply Transistor count still rising

3. Art of Multiprocessor Programming3 Still on some of your desktops: The Uniprocesor memory cpu

4. Art of Multiprocessor Programming4 In the Enterprise: The Shared Memory Multiprocessor (SMP) cache Bus shared memory cache

5. Art of Multiprocessor Programming5 Your New Desktop: The Multicore Processor (CMP) cache Bus shared memory cache All on the same chip Sun T2000 Niagara

6. Art of Multiprocessor Programming6 Multicores Are Here “Intel's Intel ups ante with 4-core chip. New microprocessor, due this year, will be faster, use less electricity...” [San Fran Chronicle] “AMD will launch a dual-core version of its Opteron server processor at an event in New York on April 21.” [PC World] “Sun’s Niagara…will have eight cores, each core capable of running 4 threads in parallel, for 32 concurrently running threads. ….” [The Inquierer]

7. Art of Multiprocessor Programming7 Why do we care? Time no longer cures software bloat –The “free ride” is over When you double your program’s path length –You can’t just wait 6 months –Your software must somehow exploit twice as much concurrency

8. Art of Multiprocessor Programming8 Traditional Scaling Process User code Traditional Uniprocessor Speedup 1.8x 7x 3.6x Time: Moore’s law

9. Art of Multiprocessor Programming9 Multicore Scaling Process User code Multicore Speedup 1.8x7x3.6x Unfortunately, not so simple…

10. Art of Multiprocessor Programming10 Real-World Scaling Process 1.8x 2x 2.9x User code Multicore Speedup Parallelization and Synchronization require great care…

11. Art of Multiprocessor Programming11 Multicore Programming: Course Overview Fundamentals –Models, algorithms, impossibility Real-World programming –Architectures –Techniques

12. Art of Multiprocessor Programming12 Multicore Programming: Course Overview Fundamentals –Models, algorithms, impossibility Real-World programming –Architectures –Techniques We don’t necessarily want to make you experts…

13. Art of Multiprocessor Programming 13 Sequential Computation memory object thread

14. Art of Multiprocessor Programming 14 Concurrent Computation memory object threads

15. Art of Multiprocessor Programming15 Asynchrony Sudden unpredictable delays –Cache misses (short) –Page faults (long) –Scheduling quantum used up (really long)

16. Art of Multiprocessor Programming16 Model Summary Multiple threads –Sometimes called processes Single shared memory Objects live in memory Unpredictable asynchronous delays

17. Art of Multiprocessor Programming17 Road Map We are going to focus on principles first, then practice –Start with idealized models –Look at simplistic problems –Emphasize correctness over pragmatism –“Correctness may be theoretical, but incorrectness has practical impact”

18. Art of Multiprocessor Programming18 Concurrency Jargon Hardware –Processors Software –Threads, processes Sometimes OK to confuse them, sometimes not.

19. Art of Multiprocessor Programming19 Parallel Primality Testing Challenge –Print primes from 1 to 10 10 Given –Ten-processor multiprocessor –One thread per processor Goal –Get ten-fold speedup (or close)

20. Art of Multiprocessor Programming20 Load Balancing Split the work evenly Each thread tests range of 10 9 … … 10 910 2·10 9 1 P0P0 P1P1 P9P9

21. Art of Multiprocessor Programming 21 Procedure for Thread i void primePrint { int i = ThreadID.get(); // IDs in {0..9} for (j = i*10 9 +1, j<(i+1)*10 9 ; j++) { if (isPrime(j)) print(j); }

22. Art of Multiprocessor Programming22 Issues Higher ranges have fewer primes Yet larger numbers harder to test Thread workloads –Uneven –Hard to predict

23. Art of Multiprocessor Programming23 Issues Higher ranges have fewer primes Yet larger numbers harder to test Thread workloads –Uneven –Hard to predict Need dynamic load balancing rejected

24. Art of Multiprocessor Programming 24 17 18 19 Shared Counter each thread takes a number

25. Art of Multiprocessor Programming 25 Procedure for Thread i int counter = new Counter(1); void primePrint { long j = 0; while (j < 10 10 ) { j = counter.getAndIncrement(); if (isPrime(j)) print(j); }

26. Art of Multiprocessor Programming 26 Counter counter = new Counter(1); void primePrint { long j = 0; while (j < 10 10 ) { j = counter.getAndIncrement(); if (isPrime(j)) print(j); } Procedure for Thread i Shared counter object

27. Art of Multiprocessor Programming27 Where Things Reside cache Bus cache 1 shared counter shared memory void primePrint { int i = ThreadID.get(); // IDs in {0..9} for (j = i*10 9 +1, j<(i+1)*10 9 ; j++) { if (isPrime(j)) print(j); } code Local variables

28. Art of Multiprocessor Programming 28 Procedure for Thread i Counter counter = new Counter(1); void primePrint { long j = 0; while (j < 10 10 ) { j = counter.getAndIncrement(); if (isPrime(j)) print(j); } Stop when every value taken

29. Art of Multiprocessor Programming 29 Counter counter = new Counter(1); void primePrint { long j = 0; while (j < 10 10 ) { j = counter.getAndIncrement(); if (isPrime(j)) print(j); } Procedure for Thread i Increment & return each new value

30. Art of Multiprocessor Programming 30 Counter Implementation public class Counter { private long value; public long getAndIncrement() { return value++; }

31. Art of Multiprocessor Programming 31 Counter Implementation public class Counter { private long value; public long getAndIncrement() { return value++; } OK for single thread, not for concurrent threads

32. Art of Multiprocessor Programming 32 What It Means public class Counter { private long value; public long getAndIncrement() { return value++; }

33. Art of Multiprocessor Programming 33 What It Means public class Counter { private long value; public long getAndIncrement() { return value++; } temp = value; value = value + 1; return temp;

34. Art of Multiprocessor Programming 34 time Not so good… Value… 1 read 1 read 1 write 2 read 2 write 3 write 2 232

35. Art of Multiprocessor Programming 35 Is this problem inherent? If we could only glue reads and writes… read write read write

36. Art of Multiprocessor Programming 36 Challenge public class Counter { private long value; public long getAndIncrement() { temp = value; value = temp + 1; return temp; }

37. Art of Multiprocessor Programming 37 Challenge public class Counter { private long value; public long getAndIncrement() { temp = value; value = temp + 1; return temp; } Make these steps atomic (indivisible)

38. Art of Multiprocessor Programming 38 Hardware Solution public class Counter { private long value; public long getAndIncrement() { temp = value; value = temp + 1; return temp; } ReadModifyWrite() instruction

39. Art of Multiprocessor Programming 39 An Aside: Java™ public class Counter { private long value; public long getAndIncrement() { synchronized { temp = value; value = temp + 1; } return temp; }

40. Art of Multiprocessor Programming 40 An Aside: Java™ public class Counter { private long value; public long getAndIncrement() { synchronized { temp = value; value = temp + 1; } return temp; } Synchronized block

41. Art of Multiprocessor Programming 41 An Aside: Java™ public class Counter { private long value; public long getAndIncrement() { synchronized { temp = value; value = temp + 1; } return temp; } Mutual Exclusion

42. Art of Multiprocessor Programming 42 Mutual Exclusion or “Alice & Bob share a pond” AB

43. Art of Multiprocessor Programming 43 Alice has a pet AB

44. Art of Multiprocessor Programming 44 Bob has a pet AB

45. Art of Multiprocessor Programming 45 The Problem AB The pets don’t get along

46. Art of Multiprocessor Programming46 Formalizing the Problem Two types of formal properties in asynchronous computation: Safety Properties –Nothing bad happens ever Liveness Properties –Something good happens eventually

47. Art of Multiprocessor Programming47 Formalizing our Problem Mutual Exclusion –Both pets never in pond simultaneously –This is a safety property No Deadlock –if only one wants in, it gets in –if both want in, one gets in. –This is a liveness property

48. Art of Multiprocessor Programming48 Simple Protocol Idea –Just look at the pond Gotcha –Trees obscure the view

49. Art of Multiprocessor Programming49 Interpretation Threads can’t “see” what other threads are doing Explicit communication required for coordination

50. Art of Multiprocessor Programming50 Cell Phone Protocol Idea –Bob calls Alice (or vice-versa) Gotcha –Bob takes shower –Alice recharges battery –Bob out shopping for pet food …

51. Art of Multiprocessor Programming51 Interpretation Message-passing doesn’t work Recipient might not be –Listening –There at all Communication must be –Persistent (like writing) –Not transient (like speaking)

52. Art of Multiprocessor Programming 52 Can Protocol cola

53. Art of Multiprocessor Programming 53 Bob conveys a bit AB cola

54. Art of Multiprocessor Programming 54 Bob conveys a bit AB cola

55. Art of Multiprocessor Programming55 Can Protocol Idea –Cans on Alice’s windowsill –Strings lead to Bob’s house –Bob pulls strings, knocks over cans Gotcha –Cans cannot be reused –Bob runs out of cans

56. Art of Multiprocessor Programming56 Interpretation Cannot solve mutual exclusion with interrupts –Sender sets fixed bit in receiver’s space –Receiver resets bit when ready –Requires unbounded number of inturrupt bits

57. Art of Multiprocessor Programming 57 Flag Protocol AB

58. Art of Multiprocessor Programming 58 Alice’s Protocol (sort of) AB

59. Art of Multiprocessor Programming 59 Bob’s Protocol (sort of) AB

60. Art of Multiprocessor Programming60 Alice’s Protocol Raise flag Wait until Bob’s flag is down Unleash pet Lower flag when pet returns

61. Art of Multiprocessor Programming61 Bob’s Protocol Raise flag Wait until Alice’s flag is down Unleash pet Lower flag when pet returns danger!

62. Art of Multiprocessor Programming62 Bob’s Protocol (2 nd try) Raise flag While Alice’s flag is up –Lower flag –Wait for Alice’s flag to go down –Raise flag Unleash pet Lower flag when pet returns

63. Art of Multiprocessor Programming63 Bob’s Protocol Raise flag While Alice’s flag is up –Lower flag –Wait for Alice’s flag to go down –Raise flag Unleash pet Lower flag when pet returns Bob defers to Alice

64. Art of Multiprocessor Programming64 The Flag Principle Raise the flag Look at other’s flag Flag Principle: –If each raises and looks, then –Last to look must see both flags up

65. Art of Multiprocessor Programming65 Proof of Mutual Exclusion Assume both pets in pond –Derive a contradiction –By reasoning backwards Consider the last time Alice and Bob each looked before letting the pets in Without loss of generality assume Alice was the last to look…

66. Art of Multiprocessor Programming 66 Proof time Alice’s last look Alice last raised her flag Bob’s last look QED Alice must have seen Bob’s Flag. A Contradiction Bob last raised flag

67. Art of Multiprocessor Programming67 Proof of No Deadlock If only one pet wants in, it gets in.

68. Art of Multiprocessor Programming68 Proof of No Deadlock If only one pet wants in, it gets in. Deadlock requires both continually trying to get in.

69. Art of Multiprocessor Programming69 Proof of No Deadlock If only one pet wants in, it gets in. Deadlock requires both continually trying to get in. If Bob sees Alice’s flag, he gives her priority (a gentleman…) QED

70. Art of Multiprocessor Programming70 Remarks Protocol is unfair –Bob’s pet might never get in Protocol uses waiting –If Bob is eaten by his pet, Alice’s pet might never get in

71. Art of Multiprocessor Programming71 Moral of Story Mutual Exclusion cannot be solved by –transient communication (cell phones) –interrupts (cans) It can be solved by – one-bit shared variables – that can be read or written

72. Art of Multiprocessor Programming72 The Arbiter Problem (an aside) Pick a point

73. Art of Multiprocessor Programming73 The Fable Continues Alice and Bob fall in love & marry

74. Art of Multiprocessor Programming74 The Fable Continues Alice and Bob fall in love & marry Then they fall out of love & divorce –She gets the pets –He has to feed them

75. Art of Multiprocessor Programming75 The Fable Continues Alice and Bob fall in love & marry Then they fall out of love & divorce –She gets the pets –He has to feed them Leading to a new coordination problem: Producer-Consumer

76. Art of Multiprocessor Programming 76 Bob Puts Food in the Pond A

77. Art of Multiprocessor Programming 77 mmm… Alice releases her pets to Feed B mmm…

78. Art of Multiprocessor Programming78 Producer/Consumer Alice and Bob can’t meet –Each has restraining order on other –So he puts food in the pond –And later, she releases the pets Avoid –Releasing pets when there’s no food –Putting out food if uneaten food remains

79. Art of Multiprocessor Programming79 Producer/Consumer Need a mechanism so that –Bob lets Alice know when food has been put out –Alice lets Bob know when to put out more food

80. Art of Multiprocessor Programming 80 Surprise Solution AB cola

81. Art of Multiprocessor Programming 81 Bob puts food in Pond AB cola

82. Art of Multiprocessor Programming 82 Bob knocks over Can AB cola

83. Art of Multiprocessor Programming 83 Alice Releases Pets AB cola yum… B

84. Art of Multiprocessor Programming 84 Alice Resets Can when Pets are Fed AB cola

85. Art of Multiprocessor Programming 85 Pseudocode while (true) { while (can.isUp()){}; pet.release(); pet.recapture(); can.reset(); } Alice’s code

86. Art of Multiprocessor Programming 86 Pseudocode while (true) { while (can.isUp()){}; pet.release(); pet.recapture(); can.reset(); } Alice’s code while (true) { while (can.isDown()){}; pond.stockWithFood(); can.knockOver(); } Bob’s code

87. Art of Multiprocessor Programming87 Correctness Mutual Exclusion –Pets and Bob never together in pond

88. Art of Multiprocessor Programming88 Correctness Mutual Exclusion –Pets and Bob never together in pond No Starvation if Bob always willing to feed, and pets always famished, then pets eat infinitely often.

89. Art of Multiprocessor Programming89 Correctness Mutual Exclusion –Pets and Bob never together in pond No Starvation if Bob always willing to feed, and pets always famished, then pets eat infinitely often. Producer/Consumer The pets never enter pond unless there is food, and Bob never provides food if there is unconsumed food. safety liveness safety

90. Art of Multiprocessor Programming 90 Could Also Solve Using Flags AB

91. Art of Multiprocessor Programming91 Waiting Both solutions use waiting –while(mumble){} Waiting is problematic –If one participant is delayed –So is everyone else –But delays are common & unpredictable

92. Art of Multiprocessor Programming92 The Fable drags on … Bob and Alice still have issues

93. Art of Multiprocessor Programming93 The Fable drags on … Bob and Alice still have issues So they need to communicate

94. Art of Multiprocessor Programming94 The Fable drags on … Bob and Alice still have issues So they need to communicate So they agree to use billboards …

95. Art of Multiprocessor Programming 95 E 1 D 2 C 3 Billboards are Large B 3 A 1 Letter Tiles From Scrabble™ box

96. Art of Multiprocessor Programming 96 E 1 D 2 C 3 Write One Letter at a Time … B 3 A 1 W 4 A 1 S 1 H 4

97. Art of Multiprocessor Programming 97 To post a message W 4 A 1 S 1 H 4 A 1 C 3 R 1 T 1 H 4 E 1 whe w

98. Art of Multiprocessor Programming 98 S 1 Let’s send another message S 1 E 1 L 1 L 1 L 1 V 4 L 1 A 1 M 3 A 1 A 1 P 3

99. Art of Multiprocessor Programming 99 Uh-Oh A 1 C 3 R 1 T 1 H 4 E 1 S 1 E 1 L 1 L 1 L 1 OK

100. Art of Multiprocessor Programming100 Readers/Writers Devise a protocol so that –Writer writes one letter at a time –Reader reads one letter at a time –Reader sees Old message or new message No mixed messages

101. Art of Multiprocessor Programming101 Readers/Writers (continued) Easy with mutual exclusion But mutual exclusion requires waiting –One waits for the other –Everyone executes sequentially Remarkably –We can solve R/W without mutual exclusion

102. Art of Multiprocessor Programming102 Why do we care? We want as much of the code as possible to execute concurrently (in parallel) A larger sequential part implies reduced performance Amdahl’s law: this relation is not linear…

103. Art of Multiprocessor Programming103 Amdahl’s Law Speedup= …of computation given n CPUs instead of 1

104. Art of Multiprocessor Programming104 Amdahl’s Law Speedup=

105. Art of Multiprocessor Programming105 Amdahl’s Law Speedup= Parallel fraction

106. Art of Multiprocessor Programming106 Amdahl’s Law Speedup= Parallel fraction Sequential fraction

107. Art of Multiprocessor Programming107 Amdahl’s Law Speedup= Parallel fraction Number of processors Sequential fraction

108. Art of Multiprocessor Programming 108 Example Ten processors 60% concurrent, 40% sequential How close to 10-fold speedup?

109. Art of Multiprocessor Programming 109 Example Ten processors 60% concurrent, 40% sequential How close to 10-fold speedup? Speedup=2.17=

110. Art of Multiprocessor Programming 110 Example Ten processors 80% concurrent, 20% sequential How close to 10-fold speedup?

111. Art of Multiprocessor Programming 111 Example Ten processors 80% concurrent, 20% sequential How close to 10-fold speedup? Speedup=3.57=

112. Art of Multiprocessor Programming 112 Example Ten processors 90% concurrent, 10% sequential How close to 10-fold speedup?

113. Art of Multiprocessor Programming 113 Example Ten processors 90% concurrent, 10% sequential How close to 10-fold speedup? Speedup=5.26=

114. Art of Multiprocessor Programming 114 Example Ten processors 99% concurrent, 01% sequential How close to 10-fold speedup?

115. Art of Multiprocessor Programming 115 Example Ten processors 99% concurrent, 01% sequential How close to 10-fold speedup? Speedup=9.17=

116. Art of Multiprocessor Programming116 The Moral Making good use of our multiple processors (cores) means Finding ways to effectively parallelize our code –Minimize sequential parts –Reduce idle time in which threads wait without

117. Art of Multiprocessor Programming117 Multicore Programming This is what this course is about… –The % that is not easy to make concurrent yet may have a large impact on overall speedup Next week: –A more serious look at mutual exclusion

118. Art of Multiprocessor Programming 118 This work is licensed under a Creative Commons Attribution- ShareAlike 2.5 License.Creative Commons Attribution- ShareAlike 2.5 License You are free: –to Share — to copy, distribute and transmit the work –to Remix — to adapt the work Under the following conditions: –Attribution. You must attribute the work to “The Art of Multiprocessor Programming” (but not in any way that suggests that the authors endorse you or your use of the work). –Share Alike. If you alter, transform, or build upon this work, you may distribute the resulting work only under the same, similar or a compatible license. For any reuse or distribution, you must make clear to others the license terms of this work. The best way to do this is with a link to –http://creativecommons.org/licenses/by-sa/3.0/. Any of the above conditions can be waived if you get permission from the copyright holder. Nothing in this license impairs or restricts the author's moral rights.