1. CSCE 411 Design and Analysis of Algorithms Background Review Prof. Jennifer Welch CSCE 411, Background Review 1

2. Outline Algorithm Analysis Tools Sorting Graph Algorithms CSCE 411, Background Review2

3. Algorithm Analysis Tools Big-Oh notation Basic probability CSCE 411, Background Review3

4. How to Measure Efficiency Machine-independent way: analyze "pseudocode" version of algorithm assume idealized machine model one instruction takes one time unit "Big-Oh" notation order of magnitude as problem size increases Worst-case analyses safe, often occurs most often, average case often just as bad CSCE 411, Background Review4

5. Faster Algorithm vs. Faster CPU A faster algorithm running on a slower machine will always win for large enough instances Suppose algorithm S1 sorts n keys in 2n 2 instructions Suppose computer C1 executes 1 billion instruc/sec When n = 1 million, takes 2000 sec Suppose algorithm S2 sorts n keys in 50nlog 2 n instructions Suppose computer C2 executes 10 million instruc/sec When n = 1 million, takes 100 sec CSCE 411, Background Review5

6. Caveat No point in finding fastest algorithm for part of the program that is not the bottleneck If program will only be run a few times, or time is not an issue (e.g., run overnight), then no point in finding fastest algorithm CSCE 411, Background Review6

7. Review of Big-Oh Notation Measure time as a function of input size and ignore multiplicative constants lower order terms f(n) = O(g(n)) : f is “≤” g f(n) = Ω(g(n)) : f is “≥” g f(n) =  (g(n)) : f is “=” g f(n) = o(g(n)) : f is “<” g f(n) =  (g(n)) : f is “>” g CSCE 411, Background Review7

8. Big-Oh f(n) = O(g(n)) means, informally, that g is an upper bound on f Formally: there exist c > 0, n 0 > 0 s.t. f(n) ≤ cg(n) for all n ≥ n 0. To show that f(n) = O(g(n)), either find c and n 0 that satisfy the definition, or use this fact: if lim n->∞ f(n)/g(n) = some nonnegative constant, then f(n) = O(g(n)). CSCE 411, Background Review8

9. Big-Omega f(n) = Ω(g(n)) means, informally, that g is an lower bound on f Formally: there exist c > 0, n 0 > 0 s.t. f(n) ≥ cg(n) for all n ≥ n 0. To show that f(n) = Ω(g(n)), either find c and n 0 that satisfy the definition, or use this fact: if lim n->∞ f(n)/g(n) = some positive constant or ∞, then f(n) = Ω(g(n)). CSCE 411, Background Review9

10. Big-Theta f(n) = Θ(g(n)) means, informally, that g is a tight bound on f Formally: there exist c 1 > 0, c 2 > 0, n 0 > 0 s.t. c 1 g(n) ≤ f(n) ≤ c 2 g(n) for all n ≥ n 0. To show that f(n) = Θ(g(n)), either find c 1, c 2, n 0 that satisfy the definition, or use this fact: if lim n->∞ f(n)/g(n) = some positive constant, then f(n) = Θ(g(n)). CSCE 411, Background Review10

11. Tying Back to Algorithm Analysis Algorithm has worst case running time O(g(n)) if there exist c > 0 and n 0 > 0 s.t. for all n ≥ n 0, every execution (including the slowest) on an input of size n takes at most cg(n) time. Algorithm has worst case running time Ω(g(n)) if there exist c > 0 and n 0 > 0 s.t. for all n ≥ n 0, at least one execution (including the slowest) on an input of size n takes at least cg(n) time. CSCE 411, Background Review11

12. More Definitions f(n) = o(g(n)) means, informally, g is a non- tight upper bound on f lim n->∞ f(n)/g(n) = 0 pronounced “little-oh” f(n) = ω(g(n)) means, informally, g is a non- tight lower bound on f lim n->∞ f(n)/g(n) = ∞ pronounced “little-omega” CSCE 411, Background Review12

13. CSCE 411, Spring 2012: Set 313 How To Solve Recurrences Ad hoc method: expand several times guess the pattern can verify with proof by induction Master theorem general formula that works if recurrence has the form T(n) = aT(n/b) + f(n) a is number of subproblems n/b is size of each subproblem f(n) is cost of non-recursive part

14. CSCE 411, Spring 2012: Set 1014 Probability Every probabilistic claim ultimately refers to some sample space, which is a set of elementary events Think of each elementary event as the outcome of some experiment Ex: flipping two coins gives sample space {HH, HT, TH, TT} An event is a subset of the sample space Ex: event "both coins flipped the same" is {HH, TT}

15. CSCE 411, Spring 2012: Set 1015 Sample Spaces and Events S A HH TH TT HT

16. CSCE 411, Spring 2012: Set 1016 Probability Distribution A probability distribution Pr on a sample space S is a function from events of S to real numbers s.t. Pr[A] ≥ 0 for every event A Pr[S] = 1 Pr[A U B] = Pr[A] + Pr[B] for every two non- intersecting ("mutually exclusive") events A and B Pr[A] is the probability of event A

17. CSCE 411, Spring 2012: Set 1017 Probability Distribution Useful facts: Pr[Ø] = 0 If A  B, then Pr[A] ≤ Pr[B] Pr[S — A] = 1 — Pr[A] // complement Pr[A U B] = Pr[A] + Pr[B] – Pr[A  B] ≤ Pr[A] + Pr[B]

18. CSCE 411, Spring 2012: Set 1018 Probability Distribution A B Pr[A U B] = Pr[A] + Pr[B] – Pr[A  B]

19. CSCE 411, Spring 2012: Set 1019 Example Suppose Pr[{HH}] = Pr[{HT}] = Pr[{TH}] = Pr[{TT}] = 1/4. Pr["at least one head"] = Pr[{HH U HT U TH}] = Pr[{HH}] + Pr[{HT}] + Pr[{TH}] = 3/4. Pr["less than one head"] = 1 — Pr["at least one head"] = 1 — 3/4 = 1/4 HH TH TT HT 1/4

20. CSCE 411, Spring 2012: Set 1020 Specific Probability Distribution discrete probability distribution: sample space is finite or countably infinite Ex: flipping two coins once; flipping one coin infinitely often uniform probability distribution: sample space S is finite and every elementary event has the same probability, 1/|S| Ex: flipping two fair coins once

21. CSCE 411, Spring 2012: Set 1021 Flipping a Fair Coin Suppose we flip a fair coin n times Each elementary event in the sample space is one sequence of n heads and tails, describing the outcome of one "experiment" The size of the sample space is 2 n. Let A be the event "k heads and n  k tails occur". Pr[A] = C(n,k)/2 n. There are C(n,k) sequences of length n in which k heads and n–k tails occur, and each has probability 1/2 n.

22. CSCE 411, Spring 2012: Set 1022 Example n = 5, k = 3 Event A is {HHHTT, HHTTH, HTTHH, TTHHH, HHTHT, HTHTH, THTHH, HTHHT, THHTH, THHHT} Pr[3 heads and 2 tails] = C(5,3)/2 5 = 10/32

23. CSCE 411, Spring 2012: Set 1023 Flipping Unfair Coins Suppose we flip two coins, each of which gives heads two-thirds of the time What is the probability distribution on the sample space? HH TH TT HT 4/9 2/9 1/9 Pr[at least one head] = 8/9

24. CSCE 411, Spring 2012: Set 1024 Independent Events Two events A and B are independent if Pr[A  B] = Pr[A]·Pr[B] I.e., probability that both A and B occur is the product of the separate probabilities that A occurs and that B occurs.

25. CSCE 411, Spring 2012: Set 1025 Independent Events Example In two-coin-flip example with fair coins: A = "first coin is heads" B = "coins are different" HH TH TT HT 1/4 A B Pr[A] = 1/2 Pr[B] = 1/2 Pr[A  B] = 1/4 = (1/2)(1/2) so A and B are independent

26. CSCE 411, Spring 2012: Set 1026 Discrete Random Variables A discrete random variable X is a function from a finite or countably infinite sample space to the real numbers. Associates a real number with each possible outcome of an experiment Define the event "X = v" to be the set of all the elementary events s in the sample space with X(s) = v. Pr["X = v"] is the sum of Pr[{s}] over all s with X(s) = v.

27. CSCE 411, Spring 2012: Set 1027 Discrete Random Variable X=v Add up the probabilities of all the elementary events in the orange event to get the probability that X = v

28. CSCE 411, Spring 2012: Set 1028 Random Variable Example Roll two fair 6-sided dice. Sample space contains 36 elementary events (1:1, 1:2, 1:3, 1:4, 1:5, 1:6, 2:1,…) Probability of each elementary event is 1/36 Define random variable X to be the maximum of the two values rolled What is Pr["X = 3"]? It is 5/36, since there are 5 elementary events with max value 3 (1:3, 2:3, 3:3, 3:2, and 3:1)

29. CSCE 411, Spring 2012: Set 1029 Independent Random Variables It is common for more than one random variable to be defined on the same sample space. E.g.: X is maximum value rolled Y is sum of the two values rolled Two random variables X and Y are independent if for all v and w, the events "X = v" and "Y = w" are independent.

30. CSCE 411, Spring 2012: Set 1030 Expected Value of a Random Variable Most common summary of a random variable is its "average", weighted by the probabilities called expected value, or expectation, or mean Definition: E[X] = ∑ v Pr[X = v] v

31. CSCE 411, Spring 2012: Set 1031 Expected Value Example Consider a game in which you flip two fair coins. You get $3 for each head but lose $2 for each tail. What are your expected earnings? I.e., what is the expected value of the random variable X, where X(HH) = 6, X(HT) = X(TH) = 1, and X(TT) = —4? Note that no value other than 6, 1, and —4 can be taken on by X (e.g., Pr[X = 5] = 0). E[X] = 6(1/4) + 1(1/4) + 1(1/4) + (—4)(1/4) = 1

32. CSCE 411, Spring 2012: Set 1032 Properties of Expected Values E[X+Y] = E[X] + E[Y], for any two random variables X and Y, even if they are not independent! E[a·X] = a·E[X], for any random variable X and any constant a. E[X·Y] = E[X]·E[Y], for any two independent random variables X and Y

33. CSCE 411, Spring 2012: Set 1033 Conditional Probability Formalizes having partial knowledge about the outcome of an experiment Example: flip two fair coins. Probability of two heads is 1/4 Probability of two heads when you already know that the first coin is a head is 1/2 Conditional probability of A given that B occurs, denoted Pr[A|B], is defined to be Pr[A  B]/Pr[B]

34. CSCE 411, Spring 2012: Set 1034 Conditional Probability A B Pr[A] = 5/12 Pr[B] = 7/12 Pr[A  B] = 2/12 Pr[A|B] = (2/12)/(7/12) = 2/7

35. CSCE 411, Spring 2012: Set 1035 Conditional Probability Definition is Pr[A|B] = Pr[A  B]/Pr[B] Equivalently, Pr[A  B] = Pr[A|B]·Pr[B]

36. Sorting Insertion Sort Heapsort Mergesort Quicksort CSCE 411, Background Review36

37. CSCE 411, Spring 2012: Set 237 Insertion Sort Review How it works: incrementally build up longer and longer prefix of the array of keys that is in sorted order take the current key, find correct place in sorted prefix, and shift to make room to insert it Finding the correct place relies on comparing current key to keys in sorted prefix Worst-case running time is  (n 2 )

38. CSCE 411, Spring 2012: Set 238 Heapsort Review How it works: put the keys in a heap data structure repeatedly remove the min from the heap Manipulating the heap involves comparing keys to each other Worst-case running time is  (n log n)

39. CSCE 411, Spring 2012: Set 239 Mergesort Review How it works: split the array of keys in half recursively sort the two halves merge the two sorted halves Merging the two sorted halves involves comparing keys to each other Worst-case running time is  (n log n)

40. CSCE 411, Spring 2012: Set 340 Mergesort Example 13242566 26451263 52641362 5264 2564 13 13 62 62 5624 56214362 1362

41. CSCE 411, Spring 2012: Set 341 Recurrence Relation for Mergesort Let T(n) be worst case time on a sequence of n keys If n = 1, then T(n) =  (1) (constant) If n > 1, then T(n) = 2 T(n/2) +  (n) two subproblems of size n/2 each that are solved recursively  (n) time to do the merge Solving recurrence gives T(n) =  (n log n)

42. CSCE 411, Spring 2012: Set 242 Quicksort Review How it works: choose one key to be the pivot partition the array of keys into those keys < the pivot and those ≥ the pivot recursively sort the two partitions Partitioning the array involves comparing keys to the pivot Worst-case running time is  (n 2 )

43. Graphs Mathematical definitions Representations adjacency list, adjacency matrix Traversals breadth-first search, depth-first search Minimum spanning trees Kruskal’s algorithm, Prim’s algorithm Single-source shortest paths Dijkstra’s algorithm, Bellman-Ford algorithm CSCE 411, Background Review43

44. Graphs Directed graph G = (V,E) V is a finite set of vertices E is the edge set, a binary relation on E (ordered pairs of vertice) In an undirected graph, edges are unordered pairs of vertices If (u,v) is in E, then v is adjacent to u and (u,v) is incident from u and incident to v; v is neighbor of u in an undirected graph, u is also adjacent to v, and (u,v) is incident on u and v Degree of a vertex is number of edges incident from or to (or on) the vertex CSCE 411, Spring 2012: Set 444

45. More on Graphs A path of length k from vertex u to vertex u’ is a sequence of k+1 vertices s.t. there is an edge from each vertex to the next in the sequence In this case, u’ is reachable from u A path is simple if no vertices are repeated A path forms a cycle if last vertex = first vertex A cycle is simple if no vertices are repeated other than first and last CSCE 411, Spring 2012: Set 445

46. Graph Representations So far, we have discussed graphs purely as mathematical concepts How can we represent a graph in a computer program? We need some data structures. Two most common representations are: adjacency list – best for sparse graphs (few edges) adjacency matrix – best for dense graphs (lots of edges) CSCE 411, Spring 2012: Set 446

47. Adjacency List Representation Given graph G = (V,E) array Adj of |V| linked lists, one for each vertex in V The adjacency list for vertex u, Adj[u], contains all vertices v s.t. (u,v) is in E each vertex in the list might be just a string, representing its name, or it might be some other data structure representing the vertex, or it might be a pointer to the data structure for that vertex Requires Θ (V+E) space (memory) Requires Θ (degree(u)) time to check if (u,v) is in E CSCE 411, Spring 2012: Set 447

48. CSCE 411, Spring 2012: Set 848 Adjacency List Representation a cd b e a b c d e bc ade ad bce bd Space-efficient for sparse graphs

49. Adjacency Matrix Representation Given graph G = (V,E) Number the vertices 1, 2,..., |V| Use a |V| x |V| matrix A with (i,j)-entry of A being 1 if (i,j) is in E 0 if (i,j) is not in E Requires Θ (V 2 ) space (memory) Requires Θ (1) time to check if (u,v) is in E CSCE 411, Spring 2012: Set 449

50. CSCE 411, Spring 2012: Set 850 Adjacency Matrix Representation a cd b e a b c d e Check for an edge in constant time a b c d e 0 1100 10011 10010 01101 01010

51. More on Graph Representations Read Ch 22, Sec 1 for more about directed vs. undirected graphs how to represented weighted graphs (some value is associated with each edge) pros and cons of adjacency list vs. adjacency matrix CSCE 411, Spring 2012: Set 451

52. CSCE 411, Spring 2012: Set 852 Graph Traversals Ways to traverse/search a graph Visit every vertex exactly once Breadth-First Search Depth-First Search

53. CSCE 411, Spring 2012: Set 853 Breadth First Search (BFS) Input: G = (V,E) and source s in V for each vertex v in V do mark v as unvisited mark s as visited enq(Q,s) // FIFO queue Q while Q is not empty do u := deq(Q) for each unvisited neighbor v of u do mark v as visited enq(Q,v)

54. CSCE 411, Spring 2012: Set 854 BFS Example start at s and consider in alphabetical order a c d b s

55. CSCE 411, Spring 2012: Set 855 BFS Tree We can make a spanning tree rooted at s by remembering the "parent" of each vertex

56. CSCE 411, Spring 2012: Set 856 Breadth First Search #2 Input: G = (V,E) and source s in V for each vertex v in V do mark v as unvisited parent[v] := nil mark s as visited parent[s] := s enq(Q,s) // FIFO queue Q

57. CSCE 411, Spring 2012: Set 857 Breadth First Search #2 while Q is not empty do u := deq(Q) for each unvisited neighbor v of u do mark v as visited parent[v] := u enq(Q,v)

58. CSCE 411, Spring 2012: Set 858 BFS Tree Example a c d b s

59. CSCE 411, Spring 2012: Set 859 BFS Trees BFS tree is not necessarily unique for a given graph Depends on the order in which neighboring vertices are processed

60. CSCE 411, Spring 2012: Set 860 BFS Numbering During the breadth-first search, assign an integer to each vertex Indicate the distance of each vertex from the source s

61. CSCE 411, Spring 2012: Set 861 Breadth First Search #3 Input: G = (V,E) and source s in V for each vertex v in V do mark v as unvisited parent[v] := nil d[v] := infinity mark s as visited parent[s] := s d[s] := 0 enq(Q,s) // FIFO queue Q

62. CSCE 411, Spring 2012: Set 862 Breadth First Search #3 while Q is not empty do u := deq(Q) for each unvisited neighbor v of u do mark v as visited parent[v] := u d[v] := d[u] + 1 enq(Q,v)

63. CSCE 411, Spring 2012: Set 863 BFS Numbering Example a c d b s d = 0 d = 1 d = 2

64. CSCE 411, Spring 2012: Set 864 Shortest Path Tree Theorem: BFS algorithm visits all and only vertices reachable from s sets d[v] equal to the shortest path distance from s to v, for all vertices v, and sets parent variables to form a shortest path tree

65. CSCE 411, Spring 2012: Set 865 BFS Running Time Initialization of each vertex takes O(V) time Every vertex is enqueued once and dequeued once, taking O(V) time When a vertex is dequeued, all its neighbors are checked to see if they are unvisited, taking time proportional to number of neighbors of the vertex, and summing to O(E) over all iterations Total time is O(V+E)

66. CSCE 411, Spring 2012: Set 866 Depth-First Search Input: G = (V,E) for each vertex u in V do mark u as unvisited for each unvisited vertex u in V do call recursiveDFS(u) recursiveDFS(u): mark u as visited for each unvisited neighbor v of u do call recursiveDFS(v)

67. CSCE 411, Spring 2012: Set 867 DFS Example start at a and consider in alphabetical order a c d b e f gh

68. CSCE 411, Spring 2012: Set 868 Disconnected Graphs What if graph is disconnected or is directed? call DFS on several vertices to visit all vertices purpose of second for-loop in non-recursive wrapper a cb d e

69. CSCE 411, Spring 2012: Set 869 DFS Tree Actually might be a DFS forest (collection of trees) Keep track of parents

70. CSCE 411, Spring 2012: Set 870 Depth-First Search #2 Input: G = (V,E) for each vertex u in V do mark u as unvisited parent[u] := nil for each unvisited vertex u in V do parent[u] := u // a root call recursive DFS(u) recursiveDFS(u): mark u as visited for each unvisited neighbor v of u do parent[v] := u call recursiveDFS(v)

71. CSCE 411, Spring 2012: Set 871 More Properties of DFS Need to keep track of more information for each vertex: discovery time: when its recursive call starts finish time: when its recursive call ends

72. CSCE 411, Spring 2012: Set 872 Depth-First Search #3 Input: G = (V,E) for each vertex u in V do mark u as unvisited parent[u] := nil time := 0 for each unvisited vertex u in V do parent[u] := u // a root call recursive DFS(u) recursiveDFS(u): mark u as visited time++ disc[u] := time for each unvisited neighbor v of u do parent[v] := u call recursiveDFS(v) time++ fin[u] := time

73. CSCE 411, Spring 2012: Set 873 Running Time of DFS initialization takes O(V) time second for loop in non-recursive wrapper considers each vertex, so O(V) iterations one recursive call is made for each vertex in recursive call for vertex u, all its neighbors are checked; total time in all recursive calls is O(E) Total time is O(V+E)

74. CSCE 411, Spring 2012: Set 474 Minimum Spanning Tree 7 16 4 5 6 8 11 15 14 17 10 13 3 12 2 9 18 Given a connected undirected graph with edge weights, find subset of edges that spans all the nodes, creates no cycle, and minimizes sum of weights

75. CSCE 411, Spring 2012: Set 475 Facts About MSTs There can be many spanning trees of a graph In fact, there can be many minimum spanning trees of a graph But if every edge has a unique weight, then there is a unique MST

76. CSCE 411, Spring 2012: Set 976 Generic MST Algorithm input: weighted undirected graph G = (V,E,w) T := empty set while T is not yet a spanning tree of G find an edge e in E s.t. T U {e} is a subgraph of some MST of G add e to T return T (as MST of G)

77. CSCE 411, Spring 2012: Set 477 Kruskal's MST algorithm 7 16 4 5 6 8 11 15 14 17 10 13 3 12 2 9 18 consider the edges in increasing order of weight, add in an edge iff it does not cause a cycle

78. CSCE 411, Spring 2012: Set 478 Implementing Kruskal's Alg. Sort edges by weight efficient algorithms known How to test quickly if adding in the next edge would cause a cycle? use disjoint set data structure, later

79. CSCE 411, Spring 2012: Set 979 Kruskal's Algorithm as a Special Case of Generic Algorithm Consider edges in increasing order of weight Add the next edge iff it doesn't cause a cycle At any point, T is a forest (set of trees); eventually T is a single tree

80. CSCE 411, Spring 2012: Set 480 Another MST Algorithm Kruskal's algorithm maintains a forest that grows until it forms a spanning tree Alternative idea is keep just one tree and grow it until it spans all the nodes Prim's algorithm At each iteration, choose the minimum weight outgoing edge to add greedy!

81. CSCE 411, Spring 2012: Set 981 Idea of Prim's Algorithm Instead of growing the MST as possibly multiple trees that eventually all merge, grow the MST from a single vertex, so that there is only one tree at any point. Also a special case of the generic algorithm: at each step, add the minimum weight edge that goes out from the tree constructed so far.

82. CSCE 411, Spring 2012: Set 982 Prim's Algorithm input: weighted undirected graph G = (V,E,w) T := empty set S := {any vertex in V} while |T| < |V| - 1 do let (u,v) be a min wt. outgoing edge (u in S, v not in S) add (u,v) to T add v to S return (S,T) (as MST of G)

83. CSCE 411, Spring 2012: Set 983 Prim's Algorithm Example a b h i c g d f e 4 87 8 11 2 76 12 4 14 9 10

84. CSCE 411, Spring 2012: Set 984 Implementing Prim's Algorithm How do we find minimum weight outgoing edge? First cut: scan all adjacency lists at each iteration. Results in O(VE) time. Try to do better.

85. CSCE 411, Spring 2012: Set 985 Implementing Prim's Algorithm Idea: have each vertex not yet in the tree keep track of its best (cheapest) edge to the tree constructed so far. To find min wt. outgoing edge, find minimum among these values use a priority queue to store the best edge info (insert and extract-min operations)

86. CSCE 411, Spring 2012: Set 986 Implementing Prim's Algorithm When a vertex v is added to T, some other vertices might have their best edges affected, but only neighbors of v add decrease-key operation to the priority queue u v TiTi T i+1 w w's best edge to T i check if this edge is cheaper for w x x's best edge to T i v's best edge to T i

87. CSCE 411, Spring 2012: Set 987 Details on Prim's Algorithm Associate with each vertex v two fields: best-wt[v] : if v is not yet in the tree, then it holds the min. wt. of all edges from v to a vertex in the tree. Initially infinity. best-node[v] : if v is not yet in the tree, then it holds the name of the vertex (node) u in the tree s.t. w(v,u) is v's best-wt. Initially nil.

88. CSCE 411, Spring 2012: Set 988 Details on Prim's Algorithm input: G = (V,E,w) // initialization initialize priority queue Q to contain all vertices, using best-wt values as keys let v 0 be any vertex in V decrease-key(Q,v 0,0) // last line means change best-wt[v 0 ] to 0 and adjust Q accordingly

89. CSCE 411, Spring 2012: Set 989 Details on Prim's Algorithm while Q is not empty do u := extract-min(Q) // vertex w/ smallest best-wt if u is not v 0 then add (u,best-node[u]) to T for each neighbor v of u do if v is in Q and w(u,v) < best-wt[v] then  best-node[v] := u  decrease-key(Q,v,w(u,v)) return (V,T) // as MST of G

90. CSCE 411, Spring 2012: Set 990 Running Time of Prim's Algorithm Depends on priority queue implementation. Let T ins be time for insert T dec be time for decrease-key T ex be time for extract-min Then we have |V| inserts and one decrease-key in the initialization: O(V  T ins +T dec ) |V| iterations of while one extract-min per iteration: O(V  T ex ) total

91. CSCE 411, Spring 2012: Set 991 Running Time of Prim's Algorithm Each iteration of while includes a for loop. Number of iterations of for loop varies, depending on how many neighbors the current vertex has Total number of iterations of for loop is O(E). Each iteration of for loop: one decrease key, so O(E  T dec ) total

92. CSCE 411, Spring 2012: Set 992 Running Time of Prim's Algorithm O(V(T ins + T ex ) + E  T dec ) If priority queue is implemented with a binary heap, then T ins = T ex = T dec = O(log V) total time is O(E log V) (Think about how to implement decrease-key in O(log V) time.)

93. CSCE 411, Spring 2012: Set 993 Shortest Paths in a Graph Let's review two important single-source shortest path algorithms: Dijkstra's algorithm Bellman-Ford algorithm

94. CSCE 411, Spring 2012: Set 994 Single Source Shortest Path Problem Given: directed or undirected graph G = (V,E,w) and source vertex s in V Find: For each t in V, a path in G from s to t with minimum weight Warning! Negative weights are a problem: s t 4 55

95. CSCE 411, Spring 2012: Set 995 Shortest Path Tree Result of a SSSP algorithm can be viewed as a tree rooted at the source Why not use breadth-first search? Works fine if all weights are the same: weight of each path is (a multiple of) the number of edges in the path Doesn't work when weights are different

96. CSCE 411, Spring 2012: Set 996 Dijkstra's SSSP Algorithm Assumes all edge weights are nonnegative Similar to Prim's MST algorithm Start with source vertex s and iteratively construct a tree rooted at s Each vertex keeps track of tree vertex that provides cheapest path from s (not just cheapest path from any tree vertex) At each iteration, include the vertex whose cheapest path from s is the overall cheapest

97. CSCE 411, Spring 2012: Set 997 Prim's vs. Dijkstra's s 5 4 1 6 Prim's MST s 5 4 1 6 Dijkstra's SSSP

98. CSCE 411, Spring 2012: Set 998 Implementing Dijkstra's Alg. How can each vertex u keep track of its best path from s? Keep an estimate, d[u], of shortest path distance from s to u Use d as a key in a priority queue When u is added to the tree, check each of u's neighbors v to see if u provides v with a cheaper path from s: compare d[v] to d[u] + w(u,v)

99. CSCE 411, Spring 2012: Set 999 Dijkstra's Algorithm input: G = (V,E,w) and source vertex s // initialization d[s] := 0 d[v] := infinity for all other vertices v initialize priority queue Q to contain all vertices using d values as keys

100. CSCE 411, Spring 2012: Set 9100 Dijkstra's Algorithm while Q is not empty do u := extract-min(Q) for each neighbor v of u do if d[u] + w(u,v) < d[v] then  d[v] := d[u] + w(u,v)  decrease-key(Q,v,d[v])  parent(v) := u

101. CSCE 411, Spring 2012: Set 9101 Dijkstra's Algorithm Example ab de c 2 8 4 9 2 4 12 10 63 source is vertex a 012345 QabcdebcdecdedededØ d[a]000000 d[b] ∞ 22222 d[c] ∞ 1210 d[d] ∞∞∞ 1613 d[e] ∞∞ 11 iteration

102. CSCE 411, Spring 2012: Set 9102 Running Time of Dijkstra's Alg. initialization: insert each vertex once O(V T ins ) O(V) iterations of while loop one extract-min per iteration => O(V T ex ) for loop inside while loop has variable number of iterations… For loop has O(E) iterations total one decrease-key per iteration => O(E T dec ) Total is O(V (T ins + T ex ) + E T dec )

103. CSCE 411, Spring 2012: Set 9103 Using Different Heap Implementations O(V(T ins + T ex ) + E  T dec ) If priority queue is implemented with a binary heap, then T ins = T ex = T dec = O(log V) total time is O(E log V) There are fancier implementations of the priority queue, such as Fibonacci heap: T ins = O(1), T ex = O(log V), T dec = O(1) (amortized) total time is O(V log V + E)

104. CSCE 411, Spring 2012: Set 9104 Using Simpler Heap Implementations O(V(T ins + T ex ) + E  T dec ) If graph is dense, so that |E| =  (V 2 ), then it doesn't help to make T ins and T ex to be at most O(V). Instead, focus on making T dec be small, say constant. Implement priority queue with an unsorted array: T ins = O(1), T ex = O(V), T dec = O(1) total is O(V 2 )

105. CSCE 411, Spring 2012: Set 9105 What About Negative Edge Weights? Dijkstra's SSSP algorithm requires all edge weights to be nonnegative even more restrictive than outlawing negative weight cycles Bellman-Ford SSSP algorithm can handle negative edge weights even "handles" negative weight cycles by reporting they exist

106. CSCE 411, Spring 2012: Set 9106 Bellman-Ford Idea Consider each edge (u,v) and see if u offers v a cheaper path from s compare d[v] to d[u] + w(u,v) Repeat this process |V| - 1 times to ensure that accurate information propgates from s, no matter what order the edges are considered in

107. CSCE 411, Spring 2012: Set 9107 Bellman-Ford SSSP Algorithm input: directed or undirected graph G = (V,E,w) //initialization initialize d[v] to infinity and parent[v] to nil for all v in V other than the source initialize d[s] to 0 and parent[s] to s // main body for i := 1 to |V| - 1 do for each (u,v) in E do // consider in arbitrary order if d[u] + w(u,v) < d[v] then d[v] := d[u] + w(u,v) parent[v] := u

108. CSCE 411, Spring 2012: Set 9108 Bellman-Ford SSSP Algorithm // check for negative weight cycles for each (u,v) in E do if d[u] + w(u,v) < d[v] then output "negative weight cycle exists"

109. CSCE 411, Spring 2012: Set 9109 Running Time of Bellman-Ford O(V) iterations of outer for loop O(E) iterations of inner for loop O(VE) time total

110. CSCE 411, Spring 2012: Set 9110 Bellman-Ford Example s c a b 3 —4 4 2 1 process edges in order (c,b) (a,b) (c,a) (s,a) (s,c)

111. CSCE 411, Spring 2012: Set 9111 Speeding Up Bellman-Ford The previous example would have converged faster if we had considered the edges in a different order in the for loop move outward from s If the graph is a DAG (no cycles), we can fully exploit this idea to speed up the running time