1 Greedy Algorithms and MST Dr. Ying Lu RAIK 283 Data Structures & Algorithms

2  Giving credit where credit is due: »Most of slides for this lecture are based on slides created by Dr. David Luebke, University of Virginia. »Some examples and slides are based on lecture notes created by Dr. Chuck Cusack, Hope College, Dr. Jim Cohoon, University of Virginia, and Dr. Ben Choi, Louisiana Technical University. »I have modified them and added new slides

3 Greedy Algorithms  Main Concept: Make the best or greedy choice at any given step.  Choices are made in sequence such that »Each individual choice is best according to some limited “short- term” criterion, that is not too expensive to evaluate »Once a choice is made, it cannot be undone! v Even if it becomes evident later that it was a poor choice v Sometimes life is like that   The goal is to »take a sequence of locally optimal actions, hoping to yield a globally optimal solution.

4 When to be Greedy  Greedy algorithms apply to problems with »Optimal substructures: optimal solutions contain optimal sub- solutions. »The greedy choice property: an optimal solution can be obtained by making the greedy choice at each step. »Many optimal solutions, but we only need one such solution.

5 Minimum Spanning Tree (MST)  A spanning tree for a connected, undirected graph, G=(V, E) is 1. a connected subgraph of G that forms an 2. undirected tree incident with each vertex.  In a weighted graph G=(V, E, W), »the weight of a subgraph is the sum of the weights of the edges in the subgraph.  A minimum spanning tree (MST) for a weighted graph is »a spanning tree with the minimum weight.

6 Minimum Spanning Tree  Problem: given a connected, undirected, weighted graph: find a spanning tree using edges that minimize the total weight

7 Minimum Spanning Tree  Which edges form the minimum spanning tree (MST) of the graph? HBC GED F A

8 Minimum Spanning Tree  Answer: HBC GED F A

9 Another Example  Given a weighted graph G=(V, E,W), find a MST of G

10 Finding a MST  MSTs satisfy the optimal substructure property: an optimal tree is composed of optimal subtrees  Principal greedy methods: algorithms by Prim and Kruskal  Prim »Grow a single tree by repeatedly adding the least cost edge that connects a vertex in the existing tree to a vertex not in the existing tree v Intermediary solution is a subtree  Kruskal »Grow a tree by repeatedly adding the least cost edge that does not introduce a cycle among the edges included so far v Intermediary solution is a spanning forest

11 Prime’s Algorithm (High-Level Pseudocode)  Prime(G) //Input: A weighted connected graph G = //Output: E T --- the set of edges composing MST of G V T = {v 0 } E T =  for i = 1 to |V| - 1 do find a minimum-weight edge e* = (u*, v*) among all the edges (u, v) such that u is in V T and v is in V-V T V T = V T  {v*} E T = E T  {e*} return E T

12 Prime’s Algorithm (High-Level Pseudocode)  Prime(G) //Input: A weighted connected graph G = //Output: E T --- the set of edges composing MST of G V T = {v 0 } E T =  for i = 1 to |V| - 1 do find a minimum-weight edge e* = (u*, v*) among all the edges (u, v) such that u is in V T and v is in V-V T V T = V T  {v*} E T = E T  {e*} return E T HBC GED F A

13 Prime’s Algorithm (High-Level Pseudocode)  Prime(G) //Input: A weighted connected graph G = //Output: E T --- the set of edges composing MST of G V T = {v 0 } E T =  for i = 1 to |V| - 1 do find a minimum-weight edge e* = (u*, v*) among all the edges (u, v) such that u is in V T and v is in V-V T V T = V T  {v*} E T = E T  {e*} return E T HBC GED F A

14 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); Grow a single tree by repeatedly adding the least cost edge that connects a vertex in the existing tree to a vertex not in the existing tree Intermediary solution is a subtree

15 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); Run on example graph

16 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v);     Run on example graph

17 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v);  0    Pick a start vertex r r

18 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v);  0    Red vertices have been removed from Q u

19 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v);  0  3  Red arrows indicate parent pointers u

20 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 14  0  3  u

21 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 14  0  3  u

22 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 14  08  3  u

23 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 10  08  3  u

24 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 10  08  3  u

25 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 102  08  3  u

26 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 102   u

27 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 102   u

28 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v);  u

29 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); u

30 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); u

31 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); u

32 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); u

33 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); u

34 Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); u

35 Review: Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); What is the hidden cost in this code?

36 Review: Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; DecreaseKey(v, w(u,v)); delete the smallest element from the min-heap decrease an element’s value in the min-heap (outline an efficient algorithm for it)

37 Review: Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; DecreaseKey(v, w(u,v)); How often is ExtractMin() called? How often is DecreaseKey() called?

38 Review: Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] =  ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); What will be the running time? There are n=|V| ExtractMin calls and m=|E| DecreaseKey calls. The priority Q implementation has a large impact on performance. E.g., O((n+m)lg n) = O(m lg n) using min-heap for Q (for connected graph, n – 1  m)

39 Finding a MST  Principal greedy methods: algorithms by Prim and Kruskal  Prim »Grow a single tree by repeatedly adding the least cost edge that connects a vertex in the existing tree to a vertex not in the existing tree v Intermediary solution is a subtree  Kruskal »Grow a tree by repeatedly adding the least cost edge that does not introduce a cycle among the edges included so far v Intermediary solution is a spanning forest

40 MST Applications? HBC GED F A

41 MST Applications  Network design »telephone, electrical power, hydraulic, TV cable, computer, road  MST gives a minimum-cost network connecting all sites  MST: the most economical construction of a network

42 Kruskal’s Algorithm (High-Level Pseudocode)  Kruskal(G) //Input: A weighted connected graph G = //Output: E T --- the set of edges composing MST of G Sort E in nondecreasing order of the edge weight E T =  ; encounter = 0 //initialize the set of tree edges and its size k = 0 //initialize the number of processed edges while encounter < |V| - 1 k = k+1 if E T  {e k } is acyclic E T = E T  {e k }; encounter = encounter + 1 return E T

43 Kruskal’s Algorithm (High-Level Pseudocode)  Kruskal(G) //Input: A weighted connected graph G = //Output: E T --- the set of edges composing MST of G Sort E in nondecreasing order of the edge weight E T =  ; encounter = 0 //initialize the set of tree edges and its size k = 0 //initialize the number of processed edges while encounter < |V| - 1 k = k+1 if E T  {e k } is acyclic E T = E T  {e k }; encounter = encounter + 1 return E T HBC GED F A

44 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } Grow a tree by repeatedly adding the least cost edge that does not introduce a cycle among the edges included so far Intermediary solution is a spanning forest

45 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } Run the algorithm:

46 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } Run the algorithm:

47 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } Run the algorithm:

48 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } ? Run the algorithm:

49 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } Run the algorithm:

50 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } 2? Run the algorithm:

51 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } Run the algorithm:

52 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } ? Run the algorithm:

53 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } Run the algorithm:

54 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } ? 21 Run the algorithm:

55 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } Run the algorithm:

56 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } ? Run the algorithm:

57 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } Run the algorithm:

58 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } ? Run the algorithm:

59 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } Run the algorithm:

60 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } ? 8 21 Run the algorithm:

61 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } Run the algorithm:

62 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } ? Run the algorithm:

63 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } 2 19? Run the algorithm:

64 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } ? Run the algorithm:

65 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } ? Run the algorithm:

66 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } Run the algorithm:

67 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } Run the algorithm:

68 Kruskal’s Algorithm Kruskal() { T =  ; for each v  V MakeSet(v); sort E by increasing edge weight w for each (u,v)  E (in sorted order) if FindSet(u)  FindSet(v) T = T U {{u,v}}; Union(FindSet(u), FindSet(v)); } What will affect the running time? Let n=|V| and m=|E| O(n) MakeSet() calls 1 Sort O(m) FindSet() calls O(n) Union() calls

69 Kruskal’s Algorithm: Running Time  To summarize: »Sort edges: O(m lg m) »O(n) MakeSet()’s »O(m) FindSet()’s »O(n) Union()’s  Upshot: »Best disjoint subsets union-find algorithm makes above 3 operations only slightly worse than linear »Refer to the textbook for a discussion on the algorithms »Overall thus O(m lg m)

70 In-Class Exercise (9.1.7)

71 In-Class Exercise  Applying Kruskal’s algorithm to (b)