1. Parallel Processing Steve Terpe CS 147

2. Overview What is Parallel Processing What is Parallel Processing Parallel Processing in Nature Parallel Processing in Nature Parallel Processing vs. Multitasking Parallel Processing vs. Multitasking Amdahl ’ s Law Amdahl ’ s Law Challenges in Parallel Processing Challenges in Parallel Processing

3. What is Parallel Processing? How to make machines solve problems better and faster? How to make machines solve problems better and faster? Physical barriers limit the extent to which single processor performance can be improved — Clock Speed & Heat Dissipation Physical barriers limit the extent to which single processor performance can be improved — Clock Speed & Heat Dissipation The next most obvious solution is to distribute the computing load among several processors The next most obvious solution is to distribute the computing load among several processors

4. What is Parallel Processing? Parallel Processing encompasses a wide variety of different things: Parallel Processing encompasses a wide variety of different things: Intel Core Duo, Quad, Cell multiprocessors, networked and distributed computer systems, SETI@Home, Folding@Home, neural nets are all examples Intel Core Duo, Quad, Cell multiprocessors, networked and distributed computer systems, SETI@Home, Folding@Home, neural nets are all examples

5. Parallel Processing in Nature The world ’ s most powerful parallel processor comes standard The world ’ s most powerful parallel processor comes standard For everything else there ’ s American Express … For everything else there ’ s American Express …

6. Parallel Processing in Nature For humans parallel processing comes easy For humans parallel processing comes easy Human Vision Human Vision Color, Motion,Depth, Shape

7. Parallel Processing in Nature Machines that are more like us, and hence more useful, will need to be able to process information more like us — in parallel. Machines that are more like us, and hence more useful, will need to be able to process information more like us — in parallel. Parallel Processing is key element in pattern recognition which distinguishes human & machine intelligence Parallel Processing is key element in pattern recognition which distinguishes human & machine intelligence This is not an easy hurdle This is not an easy hurdle

8. Parallel Processing vs. Multitasking Today, computers are great at multi-tasking, but … Today, computers are great at multi-tasking, but … Multi-tasking only creates the illusion of parallel processing Multi-tasking only creates the illusion of parallel processing The processor must switch between activities — it is only the speed with which it does so that creates the illusion of simultaneous execution The processor must switch between activities — it is only the speed with which it does so that creates the illusion of simultaneous execution The illusion is most easily shattered when running virus scan while attempting to do anything else The illusion is most easily shattered when running virus scan while attempting to do anything else

9. Parallel Processing vs. Multitasking Think about it this way: Think about it this way: Multi-tasking Multi-tasking Parallel Processing Parallel Processing

10. Amdahl’s Law Consider a single processor Consider a single processor Or two … Or two … We tend to think that 2x as much work will be done in the same time We tend to think that 2x as much work will be done in the same time Or that the same amount of work will be done in half the time Or that the same amount of work will be done in half the time

11. Amdahl’s Law Do n processors imply that a computational job should complete in 1/n time? Do n processors imply that a computational job should complete in 1/n time? Sadly, no … Sadly, no …

12. Amdahl’s Law 1967 Gene Amdahl recognizes the interrelationship of all components 1967 Gene Amdahl recognizes the interrelationship of all components Overall speedup of a system depends on the speedup of a particular component & how much that component is used Overall speedup of a system depends on the speedup of a particular component & how much that component is used S = 1/([1- f] + f / k) S = 1/([1- f] + f / k) S = overall system speed S = overall system speed f = fraction of the work performed by the faster component f = fraction of the work performed by the faster component k = speedup of new component k = speedup of new component

13. Amdahl’s Law Additionally, no matter how well (or much) you parallelize an application there will always be a small portion of work that must be done serially. Additionally, no matter how well (or much) you parallelize an application there will always be a small portion of work that must be done serially. Other processors must simply sit and wait in this interval Other processors must simply sit and wait in this interval Every algorithm has a sequential part that limits potential speedup Every algorithm has a sequential part that limits potential speedup

14. Challenges in Parallel Processing Not always obvious where to “ split ” workload or even possible. Not always obvious where to “ split ” workload or even possible. If you don ’ t use it, you lose it … programs not specifically written for parallel architecture run no more efficiently on parallel systems If you don ’ t use it, you lose it … programs not specifically written for parallel architecture run no more efficiently on parallel systems

15. Challenges in Parallel Processing Connecting your CPUs Connecting your CPUs Dynamic vs Static — connections can change from one communication to next Dynamic vs Static — connections can change from one communication to next Blocking vs Nonblocking — can simultaneous connections be present? Blocking vs Nonblocking — can simultaneous connections be present? Connections can be complete, linear, star, grid, tree, hypercube, etc. Connections can be complete, linear, star, grid, tree, hypercube, etc. Bus-based routing Bus-based routing Crossbar switching — impractical for all but the most expensive super-computers Crossbar switching — impractical for all but the most expensive super-computers 2X2 switch — can route inputs to different destinations 2X2 switch — can route inputs to different destinations

16. Challenges in Parallel Processing Dealing with memory Dealing with memory Various options: Various options: Global Shared Memory Global Shared Memory Distributed Shared Memory Distributed Shared Memory Global shared memory with separate cache for processors Global shared memory with separate cache for processors Potential Hazards: Potential Hazards: Individual CPU caches or memories can become out of synch with each other. “ Cache Coherence ” Individual CPU caches or memories can become out of synch with each other. “ Cache Coherence ” Solutions: Solutions: UMA/NUMA machines UMA/NUMA machines Snoopy cache controllers Snoopy cache controllers Write-through protocols Write-through protocols

17. Conclusion What is Parallel Processing What is Parallel Processing Parallel Processing in Nature Parallel Processing in Nature Parallel Processing vs. Multitasking Parallel Processing vs. Multitasking Amdahl ’ s Law Amdahl ’ s Law Challenges in Parallel Processing Challenges in Parallel Processing

18. Questions?