1. Spring 2006 1 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Advanced Computer Architecture Lecture 22 Distributed computer Interconnection networks

2. Spring 2006 2 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Symmetric multiprocessor (SMP) MemoryMemory... CPU Memory Controller CPU I/O Hub/Bridge Key Board Mouse Monitor BIOS EtherNet Power Supply Cooling Fan One address space, uniform access time

3. Spring 2006 3 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Coherency requirements 1.Memory operations occur in the order they were issued 2.All reads return the most current value

4. Spring 2006 4 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering MESI protocol Controller Observes/Action Bus/snoop generatedProcessor generated

5. Spring 2006 5 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering How can a SMP go faster? More cpus Faster cpus Bigger cache Optimize OS Better scheduling

6. Spring 2006 6 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Speedup vs. processors 5 10 15 20 25 30 5 10 15 20 25 30 Number of processors Speedup Ideal Actual

7. Spring 2006 7 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering What limits SMP speedup? Competing for resources: memory Bus performance Sequential program

8. Spring 2006 8 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Distributed system Goal –Connect processors (nodes) together to create fast computer –Each node works on part of the problem Node –SMP processor, cache coherent –No global or shared memory Interconnect: network for node-to-node communication

9. Spring 2006 9 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Distributed computer No global or shared memory Interconnect P M... P M P M

10. Spring 2006 10 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Distributed application Identify parallelism: often the same algorithm for each node Map data set to node ID Determine communication requirements –Synchronization –Data exchange of intermediate results –No shared data so must use message passing paradigms

11. Spring 2006 11 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Parallel programming Four key steps: done by programmer and OS

12. Spring 2006 12 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering How do you communicate fast? Auctioneer Direct bus between nodes Guess what response prior to receiving

13. Spring 2006 13 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Message passing SendRecv Time Reply Blocked Latency Request Message Reply Message P1P1 P2P2

14. Spring 2006 14 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Message latency O s = overhead to inject message L = time to traverse network O r = overhead to extract message

15. Spring 2006 15 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Interconnection network Interconnect critical to communication Interconnect P M... P M P M

16. Spring 2006 16 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Interconnect goals? Low latency High bandwidth Direct route Low cost, low power Reliable

17. Spring 2006 17 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Interconnect properties Routing distance: number of links on a route Diameter: maximum routing distance Average distance: average number of links on a route

18. Spring 2006 18 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Interconnect performance Latency (ms): overhead, routing delay, capacity, contention Bandwidth (B/s) –Total: sum of all links –Bisection: sum of links that cut the network in half

19. Spring 2006 19 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Interconnect topologies Linear array and rings Meshes and tori Trees Hypercubes

20. Spring 2006 20 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Linear array and rings Linear array Ring Ring with short links

21. Spring 2006 21 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Find the following? 1.Diameter: n/2 2.Average distance: n/4 3.Bisection BW: 2 Ring, n nodes

22. Spring 2006 22 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Mesh and torus 2-D mesh 2-D torus Both fairly easy to implement

23. Spring 2006 23 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Example mesh Paragon: 1824 nodes, 16 x 114 mesh

24. Spring 2006 24 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Find the following? Square mesh, n nodes 1.Diameter: 2(n 1/2 -1) 2.Average distance: (n 1/2 -1) 3.Bisection BW: n 1/2

25. Spring 2006 25 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Multi-dimensional meshes 6 x 3 x 2 n = 36 3 x 3 x 3 x 3, n = 81 May require long wires between nodes

26. Spring 2006 26 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering

27. Spring 2006 27 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering How can a SMP go faster? Faster clock (smaller feature size, ) More processors, 64-bit Larger, faster shared memory Larger, faster local cache memory Wider, faster, shorter shared bus

28. Spring 2006 28 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering What limits SMP speedup? Physics –IC technology (speed) –Time of flight across bus Cost –Power –Memory –Disks Key limiter

29. Spring 2006 29 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering How do you communicate fast? Low latency between nodes High bandwidth between nodes Low OS overhead

30. Spring 2006 30 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Interconnect goals? Latency as small as possible As many concurrent transfers as possible –operation bandwidth –data bandwidth Cost as low as possible

31. Spring 2006 31 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Find the following? 1.Diameter: n/2 2.Average distance: n/4 3.Bisection BW: 2 Ring, n nodes

32. Spring 2006 32 EE 437 Lillevik 437s06-l22 University of Portland School of Engineering Find the following? 1.Diameter: 2 (n 1/2 -1) 2.Average distance: n 1/2 -1 3.Bisection BW: n 1/2 Square mesh, n nodes