1. Parallel Processing and Minimum Spanning Trees Prof. Sin-Min Lee Dept. of Computer Science, San Jose State University

2. Overview What is Parallel Processing Parallel Processing in Nature Parallel Processing vs. Multitasking Amdahl’s Law Challenges in Parallel Processing

3. What is Parallel Processing? How to make machines solve problems better and faster? Physical barriers limit the extent to which single processor performance can be improved—Clock Speed & Heat Dissipation The next most obvious solution is to distribute the computing load among several processors

4. What is Parallel Processing? Parallel Processing encompasses a wide variety of different things: Intel Core Duo, Quad, Cell multiprocessors, networked and distributed computer systems, SETI@Home, Folding@Home, neural nets are all examples

5. Parallel Processing in Nature The world’s most powerful parallel processor comes standard For everything else there’s American Express…

6. Parallel Processing in Nature For humans parallel processing comes easy Human Vision Color, Motion,Depth, Shape

7. Parallel Processing in Nature Machines that are more like us, and hence more useful, will need to be able to process information more like us—in parallel. Parallel Processing is key element in pattern recognition which distinguishes human & machine intelligence This is not an easy hurdle

8. Parallel Processing vs. Multitasking Today, computers are great at multi-tasking, but… Multi-tasking only creates the illusion of parallel processing The processor must switch between activities—it is only the speed with which it does so that creates the illusion of simultaneous execution The illusion is most easily shattered when running virus scan while attempting to do anything else

9. Parallel Processing vs. Multitasking Think about it this way: Multi-tasking Parallel Processing

10. Amdahl’s Law Consider a single processor Or two… We tend to think that 2x as much work will be done in the same time Or that the same amount of work will be done in half the time

11. Amdahl’s Law Do n processors imply that a computational job should complete in 1/n time? Sadly, no…

12. Amdahl’s Law 1967 Gene Amdahl recognizes the interrelationship of all components Overall speedup of a system depends on the speedup of a particular component & how much that component is used S = 1/([1- f] + f / k) S = overall system speed f = fraction of the work performed by the faster component k = speedup of new component

13. Amdahl’s Law Additionally, no matter how well (or much) you parallelize an application there will always be a small portion of work that must be done serially. Other processors must simply sit and wait in this interval Every algorithm has a sequential part that limits potential speedup

14. Challenges in Parallel Processing Not always obvious where to “split” workload or even possible. If you don’t use it, you lose it…programs not specifically written for parallel architecture run no more efficiently on parallel systems

15. Challenges in Parallel Processing Connecting your CPUs Dynamic vs Static—connections can change from one communication to next Blocking vs Nonblocking—can simultaneous connections be present? Connections can be complete, linear, star, grid, tree, hypercube, etc. Bus-based routing Crossbar switching—impractical for all but the most expensive super- computers 2X2 switch—can route inputs to different destinations

16. Challenges in Parallel Processing Dealing with memory Various options: Global Shared Memory Distributed Shared Memory Global shared memory with separate cache for processors Potential Hazards: Individual CPU caches or memories can become out of synch with each other. “Cache Coherence” Solutions: UMA/NUMA machines Snoopy cache controllers Write-through protocols

17. In the design of electronic circuitry, it is often necessary to make the pins of several components electrically equivalent by wiring them together. To interconnect a set of n pins, we can use an arrangement of n – 1 wires, each connecting two pins. Of all such arrangements, the one that uses the least amount of wire is usually the most desirable. We can model this wiring problem with a connected, undirected graph G = (V,E), where V is the set of pins, E is the set of possible interconnections between pairs of pins, and for each edge (u, v) ∈ E, we have a weight w(u,v) specifying the cost (amount of wire needed) to connect u and v.

18. We then wish to find an acyclic subset T  E that connects all of the vertices and whose total weight is minimized. Since T is acyclic and connects all of the vertices, it must form a tree, which we call a spanning tree since it “spans” the graph G. We call the problem of determining the tree T the minimum-spanning-tree problem.

20. Undirected graph and 3 of its spanning trees Undirected Graph Spanning Trees

21. Spanning trees Suppose you have a connected undirected graph –Connected: every node is reachable from every other node –Undirected: edges do not have an associated direction...then a spanning tree of the graph is a connected subgraph in which there are no cycles A connected, undirected graph Four of the spanning trees of the graph

22. Finding a spanning tree To find a spanning tree of a graph, pick a node and call it part of the spanning tree do a search from the initial node: each time you find a node that is not in the spanning tree, add to the spanning tree both the new node and the edge you followed to get to it An undirected graphResult of a BFS starting from top Result of a DFS starting from top

23. Minimizing costs Suppose you want to supply a set of houses (say, in a new subdivision) with: –electric power –water –sewage lines –telephone lines To keep costs down, you could connect these houses with a spanning tree (of, for example, power lines) –However, the houses are not all equal distances apart To reduce costs even further, you could connect the houses with a minimum-cost spanning tree

24. Minimum-cost spanning trees Suppose you have a connected undirected graph with a weight (or cost) associated with each edge The cost of a spanning tree would be the sum of the costs of its edges A minimum-cost spanning tree is that spanning tree that has the lowest cost AB ED FC 16 19 2111 33 14 18 10 6 5 A connected, undirected graph AB ED FC 16 11 18 6 5 A minimum-cost spanning tree

37. Applications of Spanning Trees Minimal path routing in all kinds of settings –circuits, networks, roads and sewers

38. Kruskal’s algorithm Pick the cheapest edge that does not create a cycle in the tree Add the edge to the solution and remove it from the graph. Continue until all nodes are part of the tree.

39. Spanning Tree Herman Etna Old Town Bangor Hampden Orono How do we plow the fewest roads so there will always be cleared roads connecting any two towns?

40. Spanning Tree Herman Etna Old Town Bangor Hampden Orono

41. Kruskal’s Algorithm Start by initializing a graph K with all of G’s nodes and none of G’s edges. For each edge (x,y) in G (taken in increasing order of weight) –If x and y are not in same connected component Add edge (x,y) to K

42. Kruskal’s Algorithm {A,B}1 {A,D}2 {A,E}5 {D,F}5 {E,D}7 {B,C}8 {B,D}9 {C,D}12 A B C D E F 5 7 1 2 9 8 5 G A B C D E F K

43. Kruskal’s Algorithm A B C D E F K 1 {A,B}1 {A,D}2 {A,E}5 {D,F}5 {E,D}7 {B,C}8 {B,D}9 {C,D}12

44. Kruskal’s Algorithm A B C D E F K 1 2 {A,B}1 {A,D}2 {A,E}5 {D,F}5 {E,D}7 {B,C}8 {B,D}9 {C,D}12

45. Kruskal’s Algorithm A B C D E F K 1 2 5 {A,B}1 {A,D}2 {A,E}5 {D,F}5 {E,D}7 {B,C}8 {B,D}9 {C,D}12

46. Kruskal’s Algorithm A B C D E F K 1 2 5 5 {A,B}1 {A,D}2 {A,E}5 {D,F}5 {E,D}7 {B,C}8 {B,D}9 {C,D}12

47. Kruskal’s Algorithm A B C D E F K 1 2 5 5 {E,D} already connected {A,B}1 {A,D}2 {A,E}5 {D,F}5 {E,D}7 {B,C}8 {B,D}9 {C,D}12

48. Kruskal’s Algorithm A B C D E F K 1 2 5 5 {A,B}1 {A,D}2 {A,E}5 {D,F}5 {E,D}7 {B,C}8 {B,D}9 {C,D}12 8

49. Kruskal’s Algorithm {A,B}1 {A,D}2 {A,E}5 {D,F}5 {E,D}7 {B,C}8 {B,D}9 {C,D}12 A B C D E F K 1 2 5 5 8 {B,D} already connected

50. Kruskal’s Algorithm {A,B}1 {A,D}2 {A,E}5 {D,F}5 {E,D}7 {B,C}8 {B,D}9 {C,D}12 A B C D E F K 1 2 5 5 8 {C,D} already connected

51. Compare Prim and Kruskal Which one is better? Which one would you use if… –you don’t know the entire graph at the beginning? –you always want a tree in partial solutions?

54. Prim’s algorithm 1 2 6 73 5 4 10 25 22 1218 24 1416 28 1 2 6 73 5 4 10

55. Prim’s algorithm 1 2 6 73 5 4 10 25 22 1218 24 1416 28 1 2 6 73 5 4 10 25

56. 1 Prim’s algorithm 2 6 73 5 4 10 25 22 1218 24 1416 28 1 2 6 73 5 4 10 25 22 1

57. Prim’s algorithm 1 2 6 73 5 4 10 25 22 1218 24 1416 28 1 2 6 73 4 10 5 25 22 12

58. Prim’s algorithm 1 2 6 73 5 4 10 25 22 1218 24 1416 28 1 2 6 73 10 4 5 25 22 12 16

59. Prim’s algorithm 1 2 6 73 5 4 10 25 22 1218 24 1416 28 1 2 6 73 10 4 5 25 22 12 1614 Cost = 99