1. 1 Minimum Spanning Tree Problem Topic 10 ITS033 – Programming & Algorithms Asst. Prof. Dr. Bunyarit Uyyanonvara IT Program, Image and Vision Computing Lab. School of Information, Computer and Communication Technology (ICT) Sirindhorn International Institute of Technology (SIIT) Thammasat University http://www.siit.tu.ac.th/bunyarit bunyarit@siit.tu.ac.th http://www.siit.tu.ac.th/bunyarit bunyarit@siit.tu.ac.th 02 5013505 X 2005

2. 2 ITS033 Topic 01-Problems & Algorithmic Problem Solving Topic 01 - Problems & Algorithmic Problem Solving Topic 02 – Algorithm Representation & Efficiency Analysis Topic 03 - State Space of a problem Topic 04 - Brute Force Algorithm Topic 05 - Divide and Conquer Topic 06-Decrease and Conquer Topic 06 - Decrease and Conquer Topic 07 - Dynamics Programming Topic 08 - Transform and Conquer Topic 09 - Graph Algorithms Topic 10 - Minimum Spanning Tree Topic 11 - Shortest Path Problem Topic 12 - Coping with the Limitations of Algorithms Power http://www.siit.tu.ac.th/bunyarit/its033.php http://www.vcharkarn.com/vlesson/7 Midterm

3. 3 Outline Minimum Spanning Trees  Definitions  A crucial fact The Prim-Jarnik Algorithm Kruskal's Algorithm

4. 4 Minimum Spanning Tree Problem Topic 10.1 ITS033 – Programming & Algorithms Asst. Prof. Dr. Bunyarit Uyyanonvara IT Program, Image and Vision Computing Lab. School of Information, Computer and Communication Technology (ICT) Sirindhorn International Institute of Technology (SIIT) Thammasat University http://www.siit.tu.ac.th/bunyarit bunyarit@siit.tu.ac.th http://www.siit.tu.ac.th/bunyarit bunyarit@siit.tu.ac.th 02 5013505 X 2005

5. 5 Imagine: 1. You wish to connect all the computers in an office building using the least amount of cable Concrete example

6. 6 Imagine: 2. If you want to build a railway network to all provinces in Thailand with minimum cost, i.e., minimum total length of the rail way. Concrete example

7. 7

8. 8 Imagine: 3. Or a network of super- highway to connect all the cities of Bangkok with minimum cost

9. 9 Imagine: 4. We want to connect electricity to all of the houses in the whole village with minimum total of cable length Concrete example

10. 10 Each vertex in a graph G represents a computer, a province or a house Each edge represents the amount of cable, length of a railway needed to connect all vertices That‘s a weighted graph problem !! A problem of finding A Minimum Spanning Tree Minimum Spanning Tree Problem

11. 11 Given a connected, undirected graph, a spanning tree of that graph is a sub- graph which is a tree and connects all the vertices together. A single graph can have many different spanning trees. We can assign a weight to a spanning tree by computing the sum of the weights of the edges in that spanning tree. A minimum spanning tree or minimum weight spanning tree is then a spanning tree with weight less than or equal to the weight of every other spanning tree. Minimum Spanning Tree (MST) ORD PIT ATL STL DEN DFW DCA 10 1 9 8 6 3 2 5 7 4

12. 12 We are interested in: Finding a tree T that contains all the vertices of a graph G (spanning tree) and has the least total weight over all. such trees is minimum-spanning tree (MST)

13. 13 Minimum Spanning Tree

14. 14 Possible multiplicity There may be several minimum spanning trees of the same weight; in particular, if all weights are the same, every spanning tree is minimum. Properties

15. 15 Uniqueness If each edge has a distinct weight then there will only be one, unique minimum spanning tree. Minimum-cost subgraph If the weights are non-negative, then a minimum spanning tree is in fact the minimum-cost subgraph connecting all vertices, since subgraphs containing cycles necessarily have more total weight. Cycle property For any cycle C in the graph, if the weight of an edge e of C is larger than the weights of other edges of C, then this edge cannot belong to a MST. Properties

16. 16 Cycle Property Cycle Property:  Let T be a minimum spanning tree of a weighted graph G  Let e be an edge of G that is not in T and C let be the cycle formed by e with T  For every edge f of C, weight(f)  weight(e) Proof:  By contradiction  If weight(f)  weight(e) we can get a spanning tree of smaller weight by replacing e with f 8 4 2 3 6 7 7 9 8 e C f 8 4 2 3 6 7 7 9 8 C e f Replacing f with e yields a better spanning tree

17. 17 UV Partition Property Partition Property:  Consider a partition of the vertices of G into subsets U and V  Let e be an edge of minimum weight across the partition  There is a minimum spanning tree of G containing edge e Proof:  Let T be an MST of G  If T does not contain e, consider the cycle C formed by e with T and let f be an edge of C across the partition  By the cycle property, weight(f)  weight(e)  Thus, weight(f)  weight(e)  We obtain another MST by replacing f with e 7 4 2 8 5 7 3 9 8 e f 7 4 2 8 5 7 3 9 8 e f Replacing f with e yields another MST UV

18. 18 Topic 10.2 ITS033 – Programming & Algorithms Asst. Prof. Dr. Bunyarit Uyyanonvara IT Program, Image and Vision Computing Lab. School of Information, Computer and Communication Technology (ICT) Sirindhorn International Institute of Technology (SIIT) Thammasat University http://www.siit.tu.ac.th/bunyarit bunyarit@siit.tu.ac.th http://www.siit.tu.ac.th/bunyarit bunyarit@siit.tu.ac.th 02 5013505 X 2005 Prim’s Algorithm

19. 19 Try to use exhaustive search ? If we were to try an exhaustive-search approach to constructing a minimum spanning tree, we would face two serious obstacles. 1) the number of spanning trees grows exponentially with the graph size (at least for dense graphs). 2) generating all spanning trees for a given graph is not easy;

20. Greedy Technique 20 Greedy Approach The greedy approach suggests constructing a solution through a sequence of steps, each expanding a partially constructed solution obtained so far, until a complete solution to the problem is reached. On each step—and this is the central point of this technique—the choice made must be feasible, i.e., it has to satisfy the problem’s constraints. locally optimal, i.e., it has to be the best local choice among all feasible choices available on that step. irrevocable, i.e., once made, it cannot be changed on subsequent steps of the algorithm.

21. 21 Prim’s Algorithm Prim’s algorithm constructs a minimum spanning tree through a sequence of expanding subtrees. The initial subtree in such a sequence consists of a single vertex selected arbitrarily from the set V of the graph’s vertices. On each iteration, we expand the current tree in the greedy manner by simply attaching to it the nearest vertex not in that tree. The algorithm stops after all the graph’s vertices have been included in the tree being constructed.

22. 22 Prim’s Algorithm

23. 23 Prim’s Algorithm The nature of Prim’s algorithm makes it necessary to provide each vertex not in the current tree with the information about the shortest edge connecting the vertex to a tree vertex. We can provide such information by attaching two labels to a vertex: (1) The name of the nearest tree vertex and the length (the weight) of the corresponding edge. (2) Vertices that are not adjacent to any of the tree vertices canbe given the ∞ label indicating their “infinite” distance to the tree vertices and a null label for the name of the nearest tree vertex.

24. 24 Prim’s Algorithm We pick an arbitrary vertex s and we grow the MST as a cloud of vertices, starting from s We store with each vertex v a label d(v) = the smallest weight of an edge connecting v to a vertex in the cloud At each step: We add to the cloud the vertex u outside the cloud with the smallest distance label We update the labels of the vertices adjacent to u

25. 25 Prim‘s Algorithm 1.All vertices are marked as not visited 2. Any vertex v you like is chosen as starting vertex and is marked as visited (define a cluster C) 3.The smallest- weighted edge e = (v,u), which connects one vertex v inside the cluster C with another vertex u outside of C, is chosen and is added to the MST. 4.The process is repeated until a spanning tree is formed

26. 26 Example B D C A F E 7 4 2 8 5 7 3 9 8 0 7 2 8   Step 1

27. 27 Example B D C A F E 7 4 2 8 5 7 3 9 8 0 7 2 5  7 Step 2

28. 28 Example B D C A F E 7 4 2 8 5 7 3 9 8 0 7 2 5  7 Step 3

29. 29 Example B D C A F E 7 4 2 8 5 7 3 9 8 0 7 2 5 4 7 Step 4

30. 30 Example (contd.) B D C A F E 7 4 2 8 5 7 3 9 8 0 3 2 5 4 7 Step 5

31. 31 Example (contd.) B D C A F E 7 4 2 8 5 7 3 9 8 0 3 2 5 4 7 Step 6

32. 32 C FE AB D 5 64 3 4 2 12 3 2 Prim‘s Algorithm

33. 33 C FE AB D 5 64 3 4 2 12 3 2 Prim‘s Algorithm

34. 34 C FE AB D 5 64 3 4 2 12 3 2 Prim‘s Algorithm

35. 35 C FE AB D 3 4 2 12 3 2 Prim‘s Algorithm

36. 36 C FE AB D 3 2 12 3 2 Prim‘s Algorithm

37. 37 C FE AB D 3 2 12 2 3 Prim‘s Algorithm

38. 38 C FE AB D 3 2 12 2 Prim‘s Algorithm

39. 39 C FE AB D 3 2 12 2 Prim‘s Algorithm

40. 40 C FE AB D 3 2 12 2 minimum- spanning tree Prim‘s Algorithm

41. Greedy Technique 41 Prim’s Algorithm Analysis Efficiency of Prim’s algorithm depends on the data structures chosen for the graph and for the priority queue of the set V-V T whose vertex priorities are the distances to the nearest tree vertices. For example, if a graph is represented by its weight matrix and the priority queue is implemented as an unordered array, the algorithm’s running time will be in θ(|V |2).

42. Greedy Technique 42 Prim’s Algorithm A min-heap is a complete binary tree in which every element is less than or equal to its children. Deletion of the smallest element from and insertion of a new element into a min-heap of size n are O(log n) operations, and so is the operation of changing an element’s priority If a graph is represented by its adjacency linked lists and the priority queue is implemented as a min-heap, the running time of the algorithm is in O(|E| log |V |). This is because the algorithm performs |V| - 1 deletions of the smallest element and makes |E| verifications and, possibly, changes of an element’s priority in a min-heap of size not greater than |V |. Each of these operations, as noted earlier, is a O(log |V |) operation. Hence, the running time of this implementation of Prim’s algorithm is in (|V| - 1+ |E|)O(log |V |) = O(|E| log |V |)

43. 43 Prim’ Minimum Spanning Tree – Exercise

44. 44 Kruskal’s Algorithm Topic 10.3 ITS033 – Programming & Algorithms Asst. Prof. Dr. Bunyarit Uyyanonvara IT Program, Image and Vision Computing Lab. School of Information, Computer and Communication Technology (ICT) Sirindhorn International Institute of Technology (SIIT) Thammasat University http://www.siit.tu.ac.th/bunyarit bunyarit@siit.tu.ac.th http://www.siit.tu.ac.th/bunyarit bunyarit@siit.tu.ac.th 02 5013505 X 2005

45. 45 Kruskal's algorithm is an algorithm in graph theory that finds a minimum spanning tree for a connected weighted graph. If the graph is not connected, then it finds a minimum spanning forest (a minimum spanning tree for each connected component). Kruskal's algorithm is also an example of a greedy algorithm. Minimum Spanning Tree Problem

46. 46 It works as follows: 1.Each vertex is in its own cluster 2. Take the edge e with the smallest weight - if e connects two vertices in different clusters, then e is added to the MST and the two clusters, which are connected by e, are merged into a single cluster - if e connects two vertices, which are already in the same cluster, ignore it 3. Continue until n-1 edges were selected Minimum Spanning Tree Problem

47. Greedy Technique 47 9.2 Kruskal’s Algorithm

48. Greedy Technique 48 Kruskal’s Algorithm Analysis On each of its iterations, Kruskal’s algorithm has to check whether the addition of the next edge to the edges already selected would create a cycle. A new cycle is created if and only if the new edge connects two vertices already connected by a path, i.e., if and only if the two vertices belong to the same connected component Each connected component of a subgraph generated by Kruskal’s algorithm is a tree because it has no cycles.

49. Greedy Technique 49 Kruskal’s Algorithm Analysis New edge connecting two vertices may (a) or may not (b) create a cycle

50. 50 C FE AB D 5 64 3 4 2 12 3 2 Kruskal‘s Algorithm

51. 51 C FE AB D 5 64 3 4 2 12 3 2 Kruskal‘s Algorithm

52. 52 C FE AB D 5 64 3 4 2 12 3 2 Kruskal‘s Algorithm

53. 53 C FE AB D 5 64 3 4 2 12 3 2 Kruskal‘s Algorithm

54. 54 C FE AB D 5 64 3 4 2 12 3 2 Kruskal‘s Algorithm

55. 55 C FE AB D 5 64 3 4 2 12 3 2 cycle!! Kruskal‘s Algorithm

56. 56 C FE AB D 5 64 3 4 2 12 3 2 Kruskal‘s Algorithm

57. 57 C FE AB D 5 64 3 4 2 12 3 2 Kruskal‘s Algorithm

58. 58 C FE AB D 3 2 12 2 minimum- spanning tree Kruskal‘s Algorithm

59. 59 Kruskal Example JFK BOS MIA ORD LAX DFW SFO BWI PVD 867 2704 187 1258 849 144 740 1391 184 946 1090 1121 2342 1846 621 802 1464 1235 337

60. 60 Example

61. 61 Example

62. 62 Example

63. 63 Example

64. 64 Example

65. 65 Example

66. 66 Example

67. 67 Example

68. 68 Example

69. 69 Example

70. 70 Example

71. 71 Example

72. 72 Example 144 740 1391 184 946 1090 1121 2342 1846 621 802 1464 1235 337

73. Greedy Technique 73 Kruskal’s Algorithm Analysis The check whether two vertices belong to the same tree is crucial in determining running time of Kruskal’s algorithm There is efficient algorithm that perform this check => union-find algorithm With an efficient union- find algorithm, the running time of Kruskal’s algorithm will be dominated by the time needed for sorting the edge weights of a given graph. Hence, with an efficient sorting algorithm, the time efficiency of Kruskal’s algorithm will be in O(|E| log |E|).

74. 74 Data Structure for Kruskal Algortihm The algorithm maintains a forest of trees An edge is accepted it if connects distinct trees We need a data structure that maintains a partition, i.e., a collection of disjoint sets, with the operations:

75. 75 Kruskal’s Minimum Spanning Tree

76. 76

77. 77

78. 78 ITS033 Topic 01-Problems & Algorithmic Problem Solving Topic 01 - Problems & Algorithmic Problem Solving Topic 02 – Algorithm Representation & Efficiency Analysis Topic 03 - State Space of a problem Topic 04 - Brute Force Algorithm Topic 05 - Divide and Conquer Topic 06-Decrease and Conquer Topic 06 - Decrease and Conquer Topic 07 - Dynamics Programming Topic 08 - Transform and Conquer Topic 09 - Graph Algorithms Topic 10 - Minimum Spanning Tree Topic 11 - Shortest Path Problem Topic 12 - Coping with the Limitations of Algorithms Power http://www.siit.tu.ac.th/bunyarit/its033.php http://www.vcharkarn.com/vlesson/7 Midterm

79. 79 End of Chapter 10 Thank you!