1. Multiprocessors— Performance, Synchronization, Memory Consistency Models Professor Alvin R. Lebeck Computer Science 220 Fall 2001

2. 2 © Alvin R. Lebeck 2001 Message Passing Architectures Cannot directly access memory on another node IBM SP-2, Intel Paragon Cluster of workstations Interconnect CA Mem P $ CA Mem P $ Node 0 0,N-1 Node 1 0,N-1 Node 2 0,N-1 Node 3 0,N-1 CA Mem P $ CA Mem P $

3. 3 © Alvin R. Lebeck 2001 CPS 220 Small-Scale—Shared Memory Caches serve to: –Increase bandwidth versus bus/memory –Reduce latency of access –Valuable for both private data and shared data What about cache coherence? Main Memory P $ P $ P $ P $ P $ P $ P $ P $ Cache(s) and TLB 0N-1

4. 4 © Alvin R. Lebeck 2001 Coherence vs. Consistency Intuition says loads should return latest value –what is latest? Coherence concerns only one memory location Consistency concerns apparent ordering for all locations A Memory System is Coherent if –can serialize all operations to that location such that, –operations performed by any processor appear in program order »program order = order defined program text or assembly code –value returned by a read is value written by last store to that location

5. 5 © Alvin R. Lebeck 2001 CPS 220 Large Scale Shared Memory Multiprocessors Shared 100s to 1000s of nodes (processors) with single shared physical address space Use General Purpose Interconnection Network –Still have cache coherence protocol –Use messages instead of bus transactions –No hardware broadcast Communication Assist Cray T3D, T3E, Compaq EV7, SUN ES3000 Interconnect Cntrl/NI Mem P $ Cntrl/NI Mem P $ Cntrl/NI Mem P $ Cntrl/NI Mem P $

6. 6 © Alvin R. Lebeck 2001 CPS 220 Larger MPs Separate Memory per Processor Local or Remote access via memory controller Cache Coherency solution: non cached pages Alternative: directory/cache that tracks state of every block in every cache –Which caches have a copies of block, dirty vs. clean,... Info per memory block vs. per cache block? –In memory => simpler protocol (centralized/one location) –In memory => directory is ƒ(memory size) vs. ƒ(cache size) Prevent directory as bottleneck: distribute directory entries with memory, each keeping track of which Processors have copies of their blocks

7. 7 © Alvin R. Lebeck 2001 CPS 220 Directory Protocol Similar to Snoopy Protocol: 3 states –Shared: > 1 processors have data, memory up to date –Uncached –Exclusive: 1 processor(owner) has data; memory out of date In addition to cache state, must track which processors have data when in shared state Terms: –Local node is the node where a request originates –Home node is the node where the memory location of an address resides –Remote node is the node that has a copy of a cache block, whether exclusive or shared.

8. 8 © Alvin R. Lebeck 2001 CPS 220 Example Directory Protocol Message sent to directory causes 2 actions : –update the directory –more messages to satifty request Block is in Uncached state: the copy in memory is the current value, & only possible requests for that block are : –Read miss: requesting processor is sent back the data from memory and the requestor is the only sharing node. The state of the block is made Shared. –Write miss: requesting processor is sent the value and becomes the Sharing node. The block is made Exclusive to indicate that the only valid copy is cached. Sharers indicates the identity of the owner. Block is Shared, the memory value is up-to-date : –Read miss: requesting processor is sent back the data from memory & requesting processor is added to the sharing set. –Write miss: requesting processor is sent the value. All processors in the set Sharers are sent invalidate messages, & Sharers set is to identity of requesting processor. The state of the block is made Exclusive.

9. 9 © Alvin R. Lebeck 2001 CPS 220 Example Directory Protocol Block is Exclusive: current value of the block is held in the cache of the processor identified by the set Sharers (the owner), & 3 possible directory requests: –Read miss: owner processor is sent a data fetch message, which causes state of block in owner’s cache to transition to Shared and causes owner to send data to directory, where it is written to memory and sent back to the requesting processor. Identity of requesting processor is added to set Sharers, which still contains the identity of the processor that was the owner (since it still has a readable copy). –Data write-back: owner processor is replacing the block and hence must write it back. This makes the memory copy up-to-date (the home directory essentially becomes the owner), the block is now uncached, and the Sharer set is empty. –Write miss: block has a new owner. A message is sent to old owner causing the cache to send the value of the block to the directory from which it is send to the requesting processor, which becomes the new owner. Sharers is set to identity of new owner, and state of block is made Exclusive.

10. 10 © Alvin R. Lebeck 2001 CPS 220 4th C: Conflict, Capacity, Compulsory and Coherency Misses More processors: increase coherency misses while decreasing capacity misses (for fixed problem size) Cache behavior of Five Parallel Programs: –FFT Fast Fourier Transform: Matrix transposition + computation –LU factorization of dense 2D matrix (linear algebra) –Barnes-Hut n-body algorithm solving galaxy evolution probem –Ocean simluates influence of eddy & boundary currents on large- scale flow in ocean: dynamic arrays per grid Miss Rates for Snooping Protocol

11. 11 © Alvin R. Lebeck 2001 Ocean Simulate ocean currents discretize in space and time Ocean Basin

12. 12 © Alvin R. Lebeck 2001 Barnes-Hut Computing the mutual interactions of N bodies –n-body problems –stars, planets, molecules… Can approximate influence of distant bodies

13. 13 © Alvin R. Lebeck 2001 Miss Rates for Snoopy Coherence (grey) & other (white) Coherence any write miss needing upgrade Fraction of Coherence misses increases as # or processors increases (fixed size problem) Ocean has increased conflict (boundary of grid)

14. 14 © Alvin R. Lebeck 2001 Miss Rate vs. Cache Size 16 procs, 32-byte blocks Miss rate drops with increased cache size Eventually becomes entire miss rate (barnes)

15. 15 © Alvin R. Lebeck 2001 Miss Rate vs. Block size 16 procs, 64KB, two- way Generally, increasing block size reduces miss rate Barnes? False Sharing Non-shared data in shared block

16. 16 © Alvin R. Lebeck 2001 Poor Data Allocation Elements on Same Page False Sharing: Elements on Same Cache Block

17. 17 © Alvin R. Lebeck 2001 Data Blocking Elements on Same Page Elements on Same Cache Block

18. 18 © Alvin R. Lebeck 2001 CPS 220 Large Scale Shared Memory Multiprocessors Directory-based Coherence Protocol Single shared physical address space General Purpose Interconnection Network Interconnect Cntrl/NI Mem P $ Cntrl/NI Mem P $ Cntrl/NI Mem P $ Cntrl/NI Mem P $

19. 19 © Alvin R. Lebeck 2001 Directory Protocol: Miss Rate vs. # of Processors 8 to 64 processors 128KB, 64-byte blocks, 2-way Local vs. remote miss Ocean: decrease then increase –Grid structure

20. 20 © Alvin R. Lebeck 2001 Miss Ratio vs. cache size 64 procs, 64-byte blocks Local miss ratio decreases Variations in remote

21. 21 © Alvin R. Lebeck 2001 Miss Rate vs. Block Size 128-KB, 64-procs Most show decrease in both local and remote misses Barnes bathtub shape and slight increase in remote misses

22. 22 © Alvin R. Lebeck 2001 Cycles per reference vs. # of processors 128-KB, 64-byte blocks, 2-way Hit, Local Miss, Remote-home miss, Remote-3-hop miss Barnes and LU low miss rates => low average access time (barnes fig broke) FFT large 3-hop latency Ocean 64 procs, data set mapping produces conflicts.. –Some data local, some remote

23. 23 © Alvin R. Lebeck 2001 CPS 220 Synchronization Why Synchronize? Need to know when it is safe for different processes to use shared data Issues for Synchronization: –Uninterruptable instruction to fetch and update memory (atomic operation); –User level synchronization operation using this primitive; –For large scale MPs, synchronization can be a bottleneck; techniques to reduce contention and latency of synchronization

24. 24 © Alvin R. Lebeck 2001 CPS 220 Atomic Fetch and Update Memory Atomic exchange: interchange a value in a register for a value in memory –0 => synchronization variable is free –1 => synchronization variable is locked and unavailable –Set register to 1 & swap –New value in register determines success in getting lock 0 if you succeeded in setting the lock (you were first) 1 if other processor had already claimed access –Key is that exchange operation is indivisible Test-and-set: tests a value and sets it if the value passes the test Fetch-and-increment: it returns the value of a memory location and atomically increments it –0 => synchronization variable is free

25. 25 © Alvin R. Lebeck 2001 CPS 220 Atomic Fetch and Update of Memory Hard to have read & write in 1 instruction: use 2 instead Load linked (or load locked) + store conditional –Load linked returns the initial value –Store conditional returns 1 if it succeeds (no other store to same memory location since preceeding load) and 0 otherwise Example doing atomic swap with LL & SC: try:movR3,R4 ; mov exchange value llR2,0(R1); load linked scR3,0(R1); store beqzR3,try ; branch store fails movR4,R2 ; put load value in R4 Example doing fetch & increment with LL & SC: try:llR2,0(R1); load linked addiR2,R2,#1 ; increment (OK if reg–reg) scR2,0(R1) ; store beqzR2,try ; branch store fails

26. 26 © Alvin R. Lebeck 2001 CPS 220 User Level Synchronization—Operation Using this Primitive Spin locks: processor continuously tries to acquire, spinning around a loop trying to get the lock liR2,#1 lockit:exchR2,0(R1) ;atomic exchange bnezR2,lockit ;already locked? What about MP with cache coherency? –Want to spin on cache copy to avoid full memory latency –Likely to get cache hits for such variables Problem: exchange includes a write, which invalidates all other copies; this generates considerable bus traffic Solution: start by simply repeatedly reading the variable; when it changes, then try exchange (“test and test&set”): try:liR2,#1 lockit:lwR3,0(R1) ;load var bnezR3,lockit ;not free=>spin exchR2,0(R1) ;atomic exchange bnezR2,try ;already locked?

27. 27 © Alvin R. Lebeck 2001 CPS 220 Steps for Invalidate Protocol StepP0$P1$P2$Bus/Direct activity 1.Has lockShspinsShspinsShNone 2.Lock<– 0ExInvInvP0 Invalidates lock 3.ShmissShmissShWB P0; P2 gets bus 4.ShwaitsShlock = 0ShP2 cache filled 5.Shlock=0ShexchShP2 cache miss(WI) 6.InvexchInvr=0;l=1ExP2 cache filled; Inv 7.Invr=1;l=1ExlockedInvWB P2; P1 cache 8.InvspinsExInvNone

28. 28 © Alvin R. Lebeck 2001 CPS 220 For Large Scale MPs, Synchronization can be Bottleneck 20 procs spin on lock held by 1 proc, 50 cycles for bus Read miss all waiting processors to fetch lock1000 Write miss by releasing processor and invalidates50 Read miss by all waiting processors1000 Write miss by all waiting processors, one successful lock, & invalidate all copies1000 Total time for 1 proc. to acquire & release lock3050 –Each time one gets a lock, drops out of competition= 1525 –20 x 1525 = 30,000 cycles for 20 processors to pass through the lock –Problem is contention for lock and serialization of lock access: once lock is free, all compete to see who gets it Alternative: create a list of waiting processors, go through list: called a “queuing lock” –Special HW to recognize 1st lock access & lock release Another mechanism: fetch-and-increment; can be used to create barrier; wait until everyone reaches same point

29. 29 © Alvin R. Lebeck 2001 CPS 220 Memory Consistency Models What is consistency? When must a processor see the new value? e.g., seems that P1:A = 0;P2:B = 0;..... A = 1;B = 1; L1: if (B == 0)...L2:if (A == 0)... Impossible for both if statements L1 & L2 to be true? –What if write invalidate is delayed & processor continues? Memory consistency models: what are the rules for such cases? Sequential consistency: result of any execution is the same as if the accesses of each processor were kept in order and the accesses among different processors were interleaved => assignments before ifs above –SC: delay all memory accesses until all invalidates done

30. 30 © Alvin R. Lebeck 2001 CPS 220 Memory Consistency Model Schemes faster execution to sequential consistency Not really an issue for most programs; they are synchronized –A program is synchronized if all access to shared data are ordered by synchronization operations write (x)... release (s) {unlock}... acquire (s) {lock}... read(x) Only those programs willing to be non-deterministic are not synchronized Several Relaxed Models for Memory Consistency since most programs are synchronized: characterized by their attitude towards: RAR, WAR, RAW, WAW to different addresses

31. 31 © Alvin R. Lebeck 2001 Summary Multiple cooperating processors Programming Model vs. architecture Shared-Memory (small and large scale) Coherence vs. consistency Coherence protocols Performance Synchronization Consistency Models