Analysis and Design of Algorithms

Analysis and Design of Algorithms

According to math historians the true origin of the word algorism:
According to math historians the true origin of the word algorism: comes from a famous Persian author named ál-Khâwrázmî.

Khâwrázmî (780-850 A.D.) A page from his book.
Statue of Khâwrázmî in front of the Faculty of Mathematics, Amirkabir University of Technology, Tehran, Iran. A stamp issued September 6, in the Soviet Union, commemorating Khâwrázmî's 1200th birthday. A page from his book. Courtesy of Wikipedia

Computational Landscape
Design Methods: Iteration & Recursion pre/post condition, loop invariant Incremental Divide-&-Conquer Prune-&-Search Greedy Dynamic programming Randomization Reduction … Analysis Methods: Mathematical Induction Asymptotic Notation Summation Recurrence Relation Lower and Upper Bounds Adversarial argument Decision tree Recursion tree Reduction … Data Structures: List, array, stack, queue Hash table Dictionary Priority Queue Disjoint Set Union Graph … Computational Models: Random Access Machine (RAM) Turing Machine Parallel Computation Distributed Computation Quantum Computation

Algorithm An algorithm is a sequence of unambiguous instructions for solving a problem, i.e., for obtaining a required output for any legitimate input in a finite amount of time.

Analysis of Algorithms
How good is the algorithm? Correctness Time efficiency Space efficiency Does there exist a better algorithm? Lower bounds Optimality

Example Time Complexity Execution time n n log n n2 2n 1 sec 20 sec
Time complexity shows dependence of algorithm’s running time on input size. Let’s assume: Computer speed = 106 IPS, Input: a data base of size n = 106 Time Complexity Execution time n 1 sec n log n 20 sec n2 12 days 2n 40 quadrillion (1015) years

Machine Model Algorithm Analysis:
should reveal intrinsic properties of the algorithm itself. should not depend on any computing platform, programming language, compiler, computer speed, etc. Elementary steps: arithmetic: –   logic: and or not comparison:       assigning a value to a scalar variable:  ….

Complexity Space complexity Time complexity
For iterative algorithms: sums For recursive algorithms: recurrence relations

Time Complexity Time complexity shows dependence of algorithm’s running time on input size. Worst-case Average or expected-case What is it good for? Tells us how efficient our design is before its costly implementation. Reveals inefficiency bottlenecks in the algorithm. Can use it to compare efficiency of different algorithms that solve the same problem. Is a tool to figure out the true complexity of the problem itself! How fast is the “fastest” algorithm for the problem? Helps us classify problems by their time complexity.

T(n) = Q( f(n) ) T(n) = 23 n3 + 5 n2 log n + 7 n log2 n + 4 log n + 6.
drop constant multiplicative factor T(n) = 23 n n2 log n n log2 n log n drop lower order terms T(n) = Q(n3) Why do we want to do this? Asymptotically (at very large values of n) the leading term largely determines function behaviour. With a new computer technology (say, 10 times faster) the leading coefficient will change (be divided by 10). So, that coefficient is technology dependent any way! This simplification is still capable of distinguishing between important but distinct complexity classes, e.g., linear vs. quadratic, or polynomial vs exponential.

Asymptotic Notations: Q O W o w
Rough, intuitive meaning worth remembering: Theta f(n) = Q(g(n)) f(n) ≈c g(n) Big Oh f(n) = O(g(n)) f(n) ≤c g(n) Big Omega f(n) = Ω(g(n)) f(n) ≥ c g(n) Little Oh f(n) = o(g(n)) f(n) ≪c g(n) Little Omega f(n) = ω(g(n)) f(n) ≫c g(n)

limn→∞ f(n)/g(n) 0 order of growth of f(n) < order of growth of g(n) f(n)  o(g(n)), f(n)  O(g(n)) c>0 order of growth of f(n) = order of growth of g(n) f(n)  (g(n)), f(n)  O(g(n)), f(n)  (g(n)) ∞ order of growth of f(n) > order of growth of g(n) f(n)  (g(n)), f(n)  (g(n)) =

Asymptotics by ratio limit
L = lim n f(n)/g(n) If L exists, then: Theta f(n) = Q(g(n)) 0 < L <  Big Oh f(n) = O(g(n)) 0 ≤ L <  Big Omega f(n) = Ω(g(n)) 0 < L Little Oh f(n) = o(g(n)) L = 0 Little Omega f(n) = ω(g(n)) L = 

Examples: logb n vs. logc n logbn = logbc logcn
limn→∞( logbn / logcn) = limn→∞ (logbc) = logbc logbn (logcn)

L’Hôpital’s rule If limn→∞ t(n) = limn→∞ g(n) = ∞
The derivatives f´, g´ exist, Then t(n) g(n) lim n→∞ = t ´(n) g ´(n) Example: log2n vs. n

Theta : Asymptotic Tight Bound
f(n) = Q(g(n)) g(n) c1g(n) c2g(n) f(n) n0 n c1, c2, n0 >0 : n  n0 , c1g(n)  f(n)  c2g(n). +

Big Oh: Asymptotic Upper Bound
f(n) = O(g(n)) c g(n) f(n) n0 n c, n0 >0 : n  n0 , f(n)  c g(n). g(n) +

Big Omega : Asymptotic Lower Bound
f(n) = W(g(n)) g(n) c g(n) f(n) n0 n c, n0 >0 : n  n0 , cg(n)  f(n). +

Little oh : Non-tight Asymptotic Upper Bound
f(n) = o(g(n)) c g(n) f(n) n0 n c >0, n0 >0 : n  n0 , f(n) < c g(n) . No matter how small +

Little omega : Non-tight Asymptotic Lower Bound
f(n) = w(g(n)) c g(n) f(n) n0 n c >0, n0 >0 : n  n0 , f(n) > c g(n) . No matter how large +

Definitions of Asymptotic Notations
f(n) = Q(g(n)) c1,c2>0, n0>0: n  n0 , c1g(n)  f(n)  c2g(n) f(n) = O(g(n)) c>0, n0>0: n  n0 , f(n)  c g(n) f(n) = W(g(n)) c>0, n0 >0: n  n0 , c g(n)  f(n) f(n) = o(g(n)) c >0, n0>0: n  n0 , f(n) < c g(n) f(n) = w(g(n)) c >0, n0>0: n  n0 , c g(n) < f(n)

Ordering Functions << << << << <<
Factorial Constant Logarithmic << << Polynomial << Exponential << << Poly Logarithmic 5 5 log n (log n)5 n5 25n n! << << << << <<

Classifying Functions
Polynomial ? Cubic Linear Quadratic 5n 5n2 5n3 5n4

Example Problem: Sorting
Some sorting algorithms and their worst-case time complexities: Quick-Sort: Q(n2) Insertion-Sort: Q(n2) Selection-Sort: Q(n2) Merge-Sort: Q(n log n) Heap-Sort: Q(n log n) there are infinitely many sorting algorithms! So, Merge-Sort and Heap-Sort are worst-case optimal, and SORTING complexity is Q(n log n).

Theoretical analysis of time efficiency
Time efficiency is analyzed by determining the number of repetitions of the basic operation as a function of input size Basic operation: the operation that contributes most towards the running time of the algorithm. T(n) ≈ copC(n) running time execution time for basic operation Number of times basic operation is executed input size

Input size and basic operation examples
Problem Input size measure Basic operation Search for key in list of n items Number of items in list n Key comparison Multiply two matrices of floating point numbers Dimensions of matrices Floating point multiplication Compute an n Graph problem #vertices and/or edges Visiting a vertex or traversing an edge

Theoretical analysis of time efficiency
Time efficiency is analyzed by determining the number of repetitions of the basic operation as a function of input size

Best-case, average-case, worst-case
Worst case: W(n) – maximum over inputs of size n Best case: B(n) – minimum over inputs of size n Average case: A(n) – “average” over inputs of size n NOT the average of worst and best case Under some assumption about the probability distribution of all possible inputs of size n, calculate the weighted sum of expected C(n) (numbers of basic operation repetitions) over all possible inputs of size n.

Time efficiency of nonrecursive algorithms
Steps in mathematical analysis of nonrecursive algorithms: Decide on parameter n indicating input’s size Identify algorithm’s basic operation Determine worst, average, & best case for inputs of size n Set up summation for C(n) reflecting algorithm’s loop structure Simplify summation using standard formulas

Series Proof by Gauss when 9 years old (!):

General rules for sums

Some Mathematical Facts
Some mathematical equalities are:

The Execution Time of Algorithms
Each operation in an algorithm (or a program) has a cost.  Each operation takes a certain of time. count = count + 1;  take a certain amount of time, but it is constant A sequence of operations: count = count + 1; Cost: c1 sum = sum + count; Cost: c2  Total Cost = c1 + c2

The Execution Time of Algorithms (cont.)
Example: Simple If-Statement Cost Times if (n < 0) c absval = -n c else absval = n; c Total Cost <= c1 + max(c2,c3)

Example: Simple Loop Cost Times i = 1; c sum = 0; c while (i <= n) { c3 n+1 i = i + 1; c4 n sum = sum + i; c5 n } Total Cost = c1 + c2 + (n+1)*c3 + n*c4 + n*c5  The time required for this algorithm is proportional to n

Example: Nested Loop Cost Times i=1; c1 1 sum = 0; c2 1 while (i <= n) { c3 n+1 j=1; c4 n while (j <= n) { c5 n*(n+1) sum = sum + i; c6 n*n j = j + 1; c7 n*n } i = i +1; c8 n Total Cost = c1 + c2 + (n+1)*c3 + n*c4 + n*(n+1)*c5+n*n*c6+n*n*c7+n*c8  The time required for this algorithm is proportional to n2

Growth-Rate Functions – Example1
Cost Times i = 1; c sum = 0; c while (i <= n) { c3 n+1 i = i + 1; c4 n sum = sum + i; c5 n } T(n) = c1 + c2 + (n+1)*c3 + n*c4 + n*c5 = (c3+c4+c5)*n + (c1+c2+c3) = a*n + b  So, the growth-rate function for this algorithm is O(n)

Cost Times i=1; c1 1 sum = 0; c2 1 while (i <= n) { c3 n+1 j=1; c4 n while (j <= n) { c5 n*(n+1) sum = sum + i; c6 n*n j = j + 1; c7 n*n } i = i +1; c8 n T(n) = c1 + c2 + (n+1)*c3 + n*c4 + n*(n+1)*c5+n*n*c6+n*n*c7+n*c8 = (c5+c6+c7)*n2 + (c3+c4+c5+c8)*n + (c1+c2+c3) = a*n2 + b*n + c  So, the growth-rate function for this algorithm is O(n2)

Cost Times for (i=1; i<=n; i++) c1 n+1 for (j=1; j<=i; j++) c2 for (k=1; k<=j; k++) c3 x=x+1; c4 T(n) = c1*(n+1) + c2*( ) + c3* ( ) + c4*( ) = a*n3 + b*n2 + c*n + d  So, the growth-rate function for this algorithm is O(n3)

Sequential Search Unsuccessful Search:  O(n) Successful Search:
int sequentialSearch(const int a[], int item, int n){ for (int i = 0; i < n && a[i]!= item; i++); if (i == n) return –1; return i; } Unsuccessful Search:  O(n) Successful Search: Best-Case: item is in the first location of the array O(1) Worst-Case: item is in the last location of the array O(n) Average-Case: The number of key comparisons 1, 2, ..., n  O(n)

Insertion Sort an incremental algorithm
2 4 5 7 11 15 2 11 5 7 4 15 2 5 11 7 4 15 2 5 11 7 4 15 2 5 11 7 4 15 2 4 5 7 11 15

Insertion Sort: Time Complexity
Algorithm InsertionSort(A[1..n]) for i  2 .. n do LI: A[1..i –1] is sorted, A[i..n] is untouched. § insert A[i] into sorted prefix A[1..i–1] by right-cyclic-shift: key  A[i] j  i –1 while j > 0 and A[j] > key do A[j+1]  A[j] j  j –1 end-while A[j+1]  key end-for end I

Master theorem If 𝑇 𝑛 =𝑎𝑇 𝑛 𝑏 +𝑂 𝑛 𝑑 for some constants a >0, b>1, and d ≥ 0, then a < bd T(n) ∈ Θ(nd) a = bd T(n) ∈ Θ(nd lg n ) a > bd T(n) ∈ Θ(nlog b a)

The divide-and-conquer Design Paradigm
Divide the problem into subproblems. Conquer the subproblems by solving them recursively. Combine subproblem solutions. Many algorithms use this paradigm.

Divide-and-conquer Technique
a problem of size n subproblem 1 of size n/2 subproblem 2 of size n/2 a solution to subproblem 1 a solution to subproblem 2 a solution to the original problem

Divide and Conquer Examples
Sorting: mergesort and quicksort Matrix multiplication-Strassen’s algorithm Binary search Powering a Number Closest pair problem ….etc.

Binary search Find an element in a sorted array: Example: Find 9
Divide: Check middle element. Conquer: Recursively search 1 sub array. Combine: Trivial. Example: Find 9

Binary Search int binarySearch(int a[], int size, int x) { int low =0;
int high = size –1; int mid; // mid will be the index of // target when it’s found. while (low <= high) { mid = (low + high)/2; if (a[mid] < x) low = mid + 1; else if (a[mid] > x) high = mid – 1; else return mid; } return –1;

Recurrence for binary search
T(n) = 1T(n/2) + Θ(1) # subproblems cost of dividing and combining subproblem size

How much better is O(log2n)?
n O(log2n) 1024 (1KB) 16, 131, 262, 524, 1,048,576 (1MB) 1,073,741,824 (1GB) 30

Powering a Number Problem: Compute an, where n∈N. Naive algorithm:
Multiply n copies of X: X ·X ·X ··· X ·X ·X. Complexity ? Θ(n) The Spot Creativity: Is this the only and the best algorithm? Any Suggestions on using divide-and-conquer strategy?

Powering a Number Starting with X, repeatedly multiply the result by itself. We get: X, X2, X4, X8, X16, X32, … Suppose we want to compute X13. 13 = So, X13 = X8 ·X4 ·X.

Powering a Number Divide-and-conquer: Complexity: T(n) = T(n/2) + Θ(1)
T(n) = Θ(lgn)

Matrix Multiplication of n*n

Code for Matrix Multiplication
for i=1 to n for j=1 to n cij=0 for k=1 to n cij=cij + aik*bkj Running Time= ?

Matrix Multiplication by Divide-&-Conquer
= A11 A12 A21 A22 B11 B12 B21 B22 C11 C12 C21 C22

Strassen’s Idea How Strassen came up with his magic idea?
We should try get rid of as more multiplications as possible. We could do a hundred additions instead. We can reduce the numbers of subproblems in T(n)=8 * T(n/2) + Θ(n2) He must be very clever.

Strassen’s Idea Multiply 2*2 matrices with only 7 recursive multiplications. Notes: plus of matrices is commutative , but multiplication is not.

Strassen’s Algorithm 1.Divide: Partition A and B into (n/2)*(n/2) submatrices. And form terms to be multiplied using + and –. 2.Conquer: Perform 7 multiplications (P1 to P7) of (n/2)×(n/2) submatrices recursively. 3.Combine:Form C (r,s,t,u) using + and –on (n/2)×(n/2) submatrices. Write down cost of each step We got T(n)=7 * T(n/2) + Θ(n2)

Cost of Strassen’s Algorithm
T(n)=7 * T(n/2) + Θ(n2) a=7,b=2 =nlg7 case 1 T(n) = Θ(nlg7) =O(n2.81) Not so surprising? Strassen’s Algorithm is not the best. The best one has O(n2.376) (Be of theoretical interest only). But it is simple and efficient enough compared with the naïve one when n>=32.

Mergesort Algorithm: Split array A[1..n] in two and make copies of each half in arrays 𝐵[1… 𝑛 2 ] and C[1… 𝑛 2 ] Sort arrays B and C Merge sorted arrays B and C into array A

Using Divide and Conquer: Mergesort
Mergesort Strategy first last (first  last)2 Sorted Sort recursively by Mergesort Sorted Merge

Mergesort Algorithm: Split array A[1..n] in two and make copies of each half in arrays 𝐵[1… 𝑛 2 ] and C[1… 𝑛 2 ] Sort arrays B and C Merge sorted arrays B and C into array A as follows: Repeat the following until no elements remain in one of the arrays: compare the first elements in the remaining unprocessed portions of the arrays copy the smaller of the two into A, while incrementing the index indicating the unprocessed portion of that array Once all elements in one of the arrays are processed, copy the remaining unprocessed elements from the other array into A.

Algorithm: Mergesort Input: Array E and indices first and last, such that the elements E[i] are defined for first  i  last. Output: E[first], …, E[last] is a sorted rearrangement of the same elements void mergeSort(Element[] E, int first, int last) if (first < last) int mid = (first+last)/2; mergeSort(E, first, mid); mergeSort(E, mid+1, last); merge(E, first, mid, last); return;

Merge Sort How to express the cost of merge sort?
T(n) =2T(n/2)  (n) for n>1, T(1)=0  (n lg n)

cost of dividing and combining
Merge Sort 1.Divide:Trivial. 2.Conquer:Recursively sort subarrays. 3.Combine:Linear-time merge. # subproblems cost of dividing and combining subproblem size

Efficiency of Mergesort
All cases have same efficiency: Θ( n log n) Number of comparisons is close to theoretical minimum for comparison-based sorting: lg n ! ≈ n lg n n Space requirement: Θ( n ) (NOT in-place) Can be implemented without recursion (bottom-up)

Quicksort by Hoare (1962) Select a pivot (partitioning element)
Rearrange the list so that all the elements in the positions before the pivot are smaller than or equal to the pivot and those after the pivot are larger than or equal to the pivot Exchange the pivot with the last element in the first (i.e., ≤) sublist – the pivot is now in its final position Sort the two sublists recursively Apply quicksort to sort the list p A[i]≤p A[i]p

Quicksort Example Recursive implementation with the left most array entry selected as the pivot element.

Quicksort Algorithm Input: Ouput:
Array E and indices first, and last, s.t. elements E[i] are defined for first  i  last Ouput: E[first], …, E[last] is a sorted rearrangement of the array Void quickSort(Element[] E, int first, int last) if (first < last) Element pivotElement = E[first]; int splitPoint = partition(E, pivotElement, first, last); quickSort (E, first, splitPoint –1 ); quickSort (E, splitPoint +1, last ); return;

Quicksort Analysis Partition can be done in O(n) time, where n is the size of the array Let T(n) be the number of comparisons required by Quicksort If the pivot ends up at position k, then we have T(n) T(nk)  T(k 1)  n To determine best-, worst-, and average-case complexity we need to determine the values of k that correspond to these cases.

Best-Case Complexity The best case is clearly when the pivot always partitions the array equally. Intuitively, this would lead to a recursive depth of at most lg n calls We can actually prove this. In this case T(n) T(n/2)  T(n/2)  n  (n lg n)

Worst-Case and Average-Case Complexity
The worst-case is when the pivot always ends up in the first or last element. That is, partitions the array as unequally as possible. In this case T(n)  T(n1)  T(11)  n  T(n1)  n  n  (n1)  … + 1  n(n  1)/2  (n2) Average case is rather complex, but is where the algorithm earns its name. The bottom line is:

QuickSort Average-Case
S< : x < p p S> : x > p n i n – i –1 WLOG Assume: | S= | = 1. If it’s larger, it can only help! T(n) = T(i) + T(n-i-1) + Q(n), T(n) = Q(1), for n=0,1. Expected-Case: T(n) = avei { T(i) + T(n-i-1) + Q(n) : i = 0 .. n –1 } = Q(n log n)

QuickSort Average-Case
1. Multiply across by n (so that we can subsequently cancel out the summation): 2. Substitute n-1 for n: 3. Subtract (2) from (1): 4. Rearrange: 5. Divide by n(n+1) to make LHS & RHS look alike: 6. Rename: nth Harmonic number 7. Simplified recurrence: 8. Iteration: 9. Finally:

Summary of Quicksort Best case: split in the middle — Θ( n log n)
Worst case: sorted array! — Θ( n2) Average case: random arrays — Θ( n log n)

Summary of Quicksort Best case: split in the middle — Θ( n log n)
Worst case: sorted array! — Θ( n2) Average case: random arrays — Θ( n log n) Considered as the method of choice for internal sorting for large files (n ≥ 10000) Improvements: better pivot selection: median of three partitioning avoids worst case in sorted files switch to insertion sort on small subfiles elimination of recursion these combine to 20-25% improvement

Binary Heap A = a binary tree with one key per node.
Max Heap Order: A satisfies the following partial order: for every node x  root[A] : key[x]  key[parent(x)]. Full tree node allocation scheme: nodes of A are allocated in increasing order of level, and left-to-right within the same level. This allows array implementation, where array indices simulate tree pointers. 4 5 6 7 1 36 16 23 17 51 18 27 19 44 20 69 21 22 47 24 33 25 59 26 46 48 8 62 9 77 10 34 11 53 12 61 13 41 14 29 15 73 81 66 54 2 3 84 74 91

Array as Binary Heap A = 1 size[A] max size A[t] A[2t] A[2t+1]
currently unused A[t] A[2t] A[2t+1] A[ t/2 ] parent node left child right child h ≈ log n 1 n = size[A]

Some MAX Heap Properties
Root A[1] contains the maximum item. Every root-to-leaf path appears in non-increasing order. Every subtree is also max heap ordered. [Recursive structure] The key at any node is the largest among all its descendents. (x,y)  AncestorOf : A[x]  A[y], where AncestorOf = { (x,y) : node x is ancestor of node y }. L R A[1] 94 76 88 74 82 56 68 48 18 92 1 2 3 4 5 6 7 8 9 10 11 12 x y

UpHeap procedure UpHeap(A, t) § O(log n) time p  t/2 § parent of t
A[1..n] = a max-heap. Suddenly, item A[t] increases in value. Now A is a “t upward corrupted heap”: (x,y)  AncestorOf : y  t  A[x]  A[y]. Question: how would you rearrange A to make it a max-heap again? Answer: percolate A[t] up its ancestral path. procedure UpHeap(A, t) § O(log n) time Pre-Cond: A is a t upward corrupted heap Post-Cond: A is rearranged into a max-heap p  t/2 § parent of t if p = 0 or A[p]  A[t] then return A[t]  A[p] UpHeap(A,p) end t 1

UpHeap Example 94 76 88 74 82 56 68 48 18 92 1 2 3 4 5 6 7 8 9 10 11 12 94 86 88 76 82 56 68 48 18 74 92 1 2 3 4 5 6 7 8 9 10 11 12 86 94 82 88 76 86 56 68 48 18 74 92 1 2 3 4 5 6 7 8 9 10 11 12 stop

DownHeap (or Heapify) procedure DownHeap(A, t) § O(log n) time
A[1..n] = a max-heap. Suddenly, item A[t] decreases in value. Now A is a “t downward corrupted heap”: (x,y)  AncestorOf : x  t  A[x]  A[y]. Question: how would you rearrange A to make it a max-heap again? Answer: demote A[t] down along largest-child path. procedure DownHeap(A, t) § O(log n) time Pre-Cond: A is a t downward corrupted heap Post-Cond: A is rearranged into a max-heap c  2t § left child of t if c > size[A] then return § c not part of heap if c < size[A] and A[c] < A[c+1] then c  c+1 § now c is the largest child of t if A[t] < A[c] then A[t]  A[c] DownHeap(A, c) end t 2t 2t+1

DownHeap Example 94 76 88 74 82 56 68 48 18 92 1 2 3 4 5 6 7 8 9 10 11 12 94 26 88 74 76 56 68 48 18 92 1 2 3 4 5 6 7 8 9 10 11 12 26 94 74 88 26 76 56 68 48 18 92 1 2 3 4 5 6 7 8 9 10 11 12 stop

Construct Heap One application of heaps is sorting. But how do we start a heap first? Problem: Given array A[1..n], rearrange its items to form a heap. Solution: Build Incrementally: That is, make A[1..t] a max heap while incrementing t  1 .. n. That is, for t  1 .. n do UpHeap(A, t) end h = log n n i 2i 1 t Most nodes are concentrated near the bottom with larger depths but smaller heights. Idea: DownHeap is better!

Heap Construction Algorithm
Solution 3: Build Backwards on t by DownHeap(A,t). procedure ConstructHeap(A[1..n]) § O(n) time Pre-Cond: input is array A[1..n] of arbitrary numbers Post-Cond: A is rearranged into a max-heap size[A]  n § establish last node barrier LastNonleafNode  n/2 for t  LastNonleafNode downto 1 do DownHeap(A, t) end n 2i 1 t h-i h = log n L R A[1] T(n) = T(|L|) + T(|R|) + O(log n)  T(n) = O(n)

Construct Heap Example
14 62 92 26 23 56 83 12 94 88 51 42 1 2 3 4 5 6 7 8 9 10 11

14 62 92 26 23 56 83 12 94 88 51 42 1 2 3 4 5 6 7 8 9 10 11 DownHeap(A,t) t=6

14 62 92 26 23 12 83 56 94 88 51 42 1 2 3 4 5 6 7 8 9 10 11 DownHeap(A,t) t=5

14 88 92 26 23 12 83 56 94 62 51 42 1 2 3 4 5 6 7 8 9 10 11 DownHeap(A,t) t=4

14 88 92 26 23 12 83 56 51 62 94 42 1 2 3 4 5 6 7 8 9 10 11 DownHeap(A,t) t=3

14 88 42 26 23 12 83 56 51 62 94 92 1 2 3 4 5 6 7 8 9 10 11 DownHeap(A,t) t=2

14 88 42 26 94 12 23 56 51 62 83 92 1 2 3 4 5 6 7 8 9 10 11 DownHeap(A,t) t=1

MAX HEAP 94 62 42 26 88 12 23 56 51 14 83 92 1 2 3 4 5 6 7 8 9 10 11

Heap as a Priority Queue
A Priority Queue (usually implemented with some “heap” structure) is an abstract Data Structure that maintains a set S of items and supports the following operations on it: MakeEmptyHeap(S): Make an empty priory queue and call it S. ConstructHeap(S): Construct a priority queue containing the set S of items. Insert(x, S): Insert new item x into S (duplicate values allowed) DeleteMax(S): Remove and return the maximum item from S. Note: Min-Heap is used if we intend to do DeleteMin instead of DeleteMax.

Priority Queue Operations
Array A as a binary heap is a suitable implementation. For a heap of size n, it has the following time complexities: O(1) MakeEmptyHeap(A) size[A]  0 O(n) ConstructHeap(A[1..n]) discussed already O(log n) Insert(x,A) and DeleteMax(A) see below procedure Insert(x, A) size[A]  size[A] + 1 A[ size[A] ]  x UpHeap(A, size[A] ) end size[A] 1 procedure DeleteMax(A) if size[A] = 0 then return error MaxItem A[1] A[1]  A[size[A]] size[A]  size[A] – 1 DownHeap(A, 1) return MaxItem end

HeapSort Algorithm HeapSort(A[1..n]) § O(n log n) time
Pre-Cond: input is array A[1..n] of arbitrary numbers Post-Cond: A is rearranged into sorted order ConstructMaxHeap(A[1..n]) for t  n downto 2 do A[t]  DeleteMax(A) end

Analysis of Heapsort (continued)
Recall algorithm: Build heap Remove root –exchange with last (rightmost) leaf Fix up heap (excluding last leaf) Repeat 2, 3 until heap contains just one node. Θ(n) Θ(log k) k=n – 1, n-2, … 1 Total: Θ(n) + Θ( n log n) = Θ(n log n) Note: this is the worst case. Average case also Θ(n log n).

HeapSort Example already constructed heap 1 2 3 4 5 6 7 8 9 10 11 12
94 76 88 74 82 56 68 48 18 92 1 2 3 4 5 6 7 8 9 10 11 12 94 | 82 | 92 | 74 | 76 | 68 | 88 | 68 | 48 | 74 | 18 | 56 already constructed heap size[A] = 12 swap Max Item

HeapSort Example 56 | 82 | 92 | 74 | 76 | 68 | 88 | 68 | 48 | 74 | 18 | 94 size[A] = 12 56 76 88 74 82 94 68 48 18 92 1 2 3 4 5 6 7 8 9 10 11 12

HeapSort Example 56 | 82 | 92 | 74 | 76 | 68 | 88 | 68 | 48 | 74 | 18 | 94 size[A] = 11 56 76 88 74 82 94 68 48 18 92 1 2 3 4 5 6 7 8 9 10 11 12 DownHeap(A,1)

HeapSort Example 92 | 82 | 88 | 74 | 76 | 68 | 56 | 68 | 48 | 74 | 18 | 94 swap Max Item size[A] = 11 1 92 2 3 82 88 5 4 6 7 74 76 68 56 68 48 74 18 94 8 9 10 11 12

HeapSort Example 74 | 82 | 68 | 74 | 76 | 18 | 56 | 68 | 48 | 88 | 92 | 94 size[A] = 10 1 74 2 3 82 68 5 4 6 7 74 76 18 56 68 48 88 92 94 8 9 10 11 12

HeapSort Example 74 | 68 | 68 | 56 | 48 | 18 | 74 | 76 | 82 | 88 | 92 | 94 swap Max Item size[A] = 6 74 48 88 68 94 76 18 82 92 56 1 2 3 4 5 6 7 8 9 10 11 12

HeapSort Example 18 | 48 | 56 | 68 | 68 | 74 | 74 | 76 | 82 | 88 | 92 | 94 size[A] = 1 SORTED ARRAY 1 18 2 3 48 56 5 4 6 7 68 68 74 74 76 82 88 92 94 8 9 10 11 12

A word on Priority Queues
In many priority queue applications, an item and the priority of the item are two distinct object types. For instance: items are vertices in a graph, while priority of an item is the shortest distance from source to the corresponding vertex in the graph. We store (pointers to) items in the nodes of the heap, while item priorities are stored separately, external to the heap. We also maintain two-way “pointers” (node[v] and v) between items and their heap node location for direct access both ways. (node[v]=0 means v is not in the heap.) Heap ordering property is based not on the items but their priorities. HEAP priority d[v] v GRAPH t node[v] = t

SORTING LOWER BOUND

n Black Boxes Box 1 Box 2 Box 3 The adversary shows you 3 black boxes.
Each contains a distinct number. Your task is to order these boxes in increasing order of the numbers they contain. You are not allowed to open the boxes or examine their contents in any way. You can only ask questions of the type: “Is the number in Box x < the number in Box y?” ( “x : y” for short) where x and y are in {1, 2, 3} and completely your choice. In the worst case, how many questions of this type do you need to ask? Below is a possible scenario called the decision tree of the strategy you might use: 1,3,2 3,1,2 2,1,3 2,3,1 1 : 3 1,2,3 3,2,1 2 : 3 1 : 2 < >

Sorting Lower Bound:  log n!  W(n log n) log n!
worst-path down log 1 = 0 Sorting Lower Bound: length of worst path down the decision tree is  log n!  W(n log n)

Multiplication of large integers
a , b are both n-digit integers If we use the brute-force approach to compute c = a * b, what is the time efficiency?

n-bit Integer Addition vs Multiplication
Proof: A correct algorithm must “look” at every input bit. Suppose on non-zero inputs, input bit b is not looked at by the algorithm. Adversary gives the algorithm the same input, but with bit b flipped. Algorithm is oblivious to b, so it will give the same answer. It can’t be correct both times! Elementary School Addition Algorithm has O(n) bit-complexity: XXXXXXXXXX + XXXXXXXXXX XXXXXXXXXXX The bit-complexity of n-bit addition is Q(n).

n-bit Integer Multiplication
Elementary School Multiplication Algorithm has O(n2) bit-complexity: XXXX * XXXX XXXXX XXXXX XXXXX XXXXX XXXXXXXXX

Multiplication of large integers (divide-conquer recursive algorithm I ]
a = a1a0 and b = b1b0 c = a * b = (a110n/2 + a0) * (b110n/2 + b0) =(a1 * b1)10n + (a1 * b0 + a0 * b1)10n/2 + (a0 * b0)

Example X = 3141 Y = 5927 X*Y = 18,616,707 X = = a1 • a0 Y = = b1 • b0 X*Y =(a1 * b1) • (a1 *b0 + a0 * b1) • a0 * b0 = (31*59) • (31* *59) • *27 = 1829 • ( ) • = • • = = 18,616,707

Multiplication of large integers (divide-conquer recursive algorithm II ]
a = a1a0 and b = b1b0 c = a * b =(a1 * b1)10n + (a1 * b0 + a0 * b1)10n/2 + (a0 * b0) =c210n + c110n/2 + c0, where c2 = a1 * b1 is the product of their first halves c0 = a0 * b0 is the product of their second halves c1 = (a1 + a0) * (b1 + b0) – (c2 + c0) is the product of the sum of the a’s halves and the sum of the b’s halves minus the sum of c2 and c0.

X = 3141 Y = 5927 X*Y = 18,616,707 X = = a1 • a0 Y = = b1 • b0 c1 = (a1 + a0)*(b1 + b0) = (31+41)*(59+27) = 72 * 86 = 6,192 c2 = a1 * b1 = 31 * 59 = 1,829 c0 = a0 * b0 = 41 * 27 = 1,107 X*Y = c2 • (c1 – c2 – c0) • c0 = 1829 • (6192 –1829 –1107) • = 18,616,707

Multiplication of large integers

Complexity of n-bit Integer Multiplication
Known Upper Bounds: O(n log 3 ) = O(n1.59 ) by divide-&-conquer [Karatsuba-Ofman, 1962] O(n log n log log n) by FFT [Schönhage-Strassen, 1971] n log n 2O(log*n) [Martin Fürer, 2007] Known Lower Bound: W( n log n / log log n) [Fischer-Meyer, 1974]

EXAMPLE: tower of hanoi
Problem: Given three pegs (A, B, C) and n disks of different sizes Initially, all the disks are on peg A in order of size, the largest on the bottom and the smallest on top The goal is to move all the disks to peg C using peg B as an auxiliary Only 1 disk can be moved at a time, and a larger disk cannot be placed on top of a smaller one B n disks A C

Design a recursive algorithm to solve this problem: Given three pegs (A, B, C) and n disks of different sizes Initially, all the disks are on peg A in order of size, the largest on the bottom and the smallest on top The goal is to move all the disks to peg C using peg B as an auxiliary Only 1 disk can be moved at a time, and a larger disk cannot be placed on top of a smaller one B n disks A C

Solve simple case when n<=1? Just trivial B B A C A C Move(A, C)

B TOWER(n-1, A, C, B) B A C A C B Move(A, C) TOWER(n, A, B, C) A C TOWER(n-1, B, A, C) B A C

TOWER(n, A, B, C) { TOWER(n-1, A, C, B); Move(A, C); TOWER(n-1, B, A, C) } if n<1 return;

Growth-Rate Functions – Recursive Algorithms
void hanoi(int n, char source, char dest, char spare) { Cost if (n > 0) { c1 hanoi(n-1, source, spare, dest); c2 cout << "Move top disk from pole " << source c3 << " to pole " << dest << endl; hanoi(n-1, spare, dest, source); c4 } } The time-complexity function T(n) of a recursive algorithm is defined in terms of itself, and this is known as recurrence equation for T(n). To find the growth-rate function for a recursive algorithm, we have to solve its recurrence relation.

Growth-Rate Functions – Hanoi Towers
What is the cost of hanoi(n,’A’,’B’,’C’)? when n=0 T(0) = c1 when n>0 T(n) = c1 + c2 + T(n-1) + c3 + c4 + T(n-1) = 2*T(n-1) + (c1+c2+c3+c4) = 2*T(n-1) + c  recurrence equation for the growth-rate function of hanoi-towers algorithm Now, we have to solve this recurrence equation to find the growth-rate function of hanoi-towers algorithm

Growth-Rate Functions – Hanoi Towers (cont.)
There are many methods to solve recurrence equations, but we will use a simple method known as repeated substitutions. T(n) = 2*T(n-1) + c = 2 * (2*T(n-2)+c) + c = 2 * (2* (2*T(n-3)+c) + c) + c = 23 * T(n-3) + ( )*c (assuming n>2) when substitution repeated i-1th times = 2i * T(n-i) + (2i )*c when i=n = 2n * T(0) + (2n )*c = 2n * c1 + ( )*c = 2n * c1 + ( 2n-1 )*c = 2n*(c1+c) – c  So, the growth rate function is O(2n)

Example: Fibonacci numbers Method 1: the recurrence formula
F0 =0, F1 =1, Fn+2 = Fn+1 + Fn for all n  0. n: … Fn : …

Example: Fibonacci numbers
Computing the nth Fibonacci number recursively (top-down): f(n) f(n-1) f(n-2) f(n-2) f(n-3) f(n-3) f(n-4) ... Algorithm Fib(n) if n{0,1} then return n return Fib(n-1) + Fib(n-2) end

Example: Fibonacci numbers

Example: Fibonacci numbers Method 2: Iterative
Computing the nth Fibonacci number using bottom-up iteration: f(0) = 0 f(1) = 1 f(2) = 0+1 = 1 f(3) = 1+1 = 2 f(4) = 1+2 = 3 f(5) = 2+3 = 5 f(n-2) = f(n-1) = f(n) = f(n-1) + f(n-2) Algorithm Fib(n) (x,y)  (0,1) § (x,y)= (F0,F1 ) for t  2 .. n do z  x + y x  y y  z LI: (x, y) = (Ft-1, Ft ) end-for return y end Cons: This is exponential time in the input n’s bit length!

Method 3: recursive doubling
Algorithm Fib(n) X EXP( (1 +5)/2 , n) Y EXP( (1 –5)/2 , n) return (X –Y)/5 end Pros: This is linear in the input n’s bit length! Cons: What if we didn’t know the formula? It uses non-integer real arithmetic &  even though Fn is always an integer! Is there a way to do integer-only arithmetic to avoid “round off” errors?

Method 4: An recursive doubling
G(n) G(n-1) A G(n) = A · G(n-1) = A2 · G(n-2) = ··· = An-1 · G(1) = An . EXP(A , n) on 2-by-2 matrices! O(log n) time integer-only arithmetic!

Memoization: Recursive vs Iterative
Algorithm Fib(n) § recursive top-down for t  do Memo[t]  t for t  2 .. n do Memo[t]  null return F(n) end Function F(n) if Memo[n] = null then Memo[n]  F(n-1) + F(n-2) return Memo[n] F(n-2) F(n) F(n-1) F(n-3) F(n-4) F(2) F(1) F(0) F(n-5) Algorithm Fib(n) § iterative bottom-up (from smallest to largest) for t  do Memo[t]  t for t  2 .. n do Memo[t]  Memo[t-1] + Memo[t-2] return Memo[n] end

Dynamic programming Dynamic Programming is a general algorithm design technique Invented by American mathematician Richard Bellman in the 1950s to solve optimization problems Main idea: set up a recurrence relating a solution to a larger instance to solutions of some smaller instances solve smaller instances once record solutions in a table extract solution to the initial instance from that table

Examples of dynamic programming algorithms
Coin-row problem Computing binomial coefficients Longest Common Subsequence (LCS) Some instances of difficult discrete optimization problems:0-1 knapsack etc…

Coin-row problem There is a row of n coins whose values are some positive integers c₁, c₂,...,cn, not necessarily distinct. The goal is to pick up the maximum amount of money subject to the constraint that no two coins adjacent in the initial row can be picked up. E.g.: 5, 1, 2, 10, 6, 2, the best selection is 5, 10, 2, giving the maximum amount of 17.

DP solution to the coin-row problem
Let F(n) be the maximum amount that can be picked up from the row of n coins. To derive a recurrence for F(n), we partition all the allowed coin selections into two groups: those without last coin – the max amount is ? those with the last coin -- the max amount is ? Thus we have the following recurrence F(n) = max{cn + F(n-2), F(n-1)} for n > 1, F(0) = 0, F(1)=c₁

DP solution to the coin-row problem (cont.)
F(n) = max{cn + F(n-2), F(n-1)} for n > 1, F(0) = 0, F(1)=c₁ index 1 2 3 4 5 6 coins -- 10 F( ) Max amount: Coins of optimal solution: Time efficiency: Space efficiency: Note: All smaller instances were solved.

Computing a binomial coefficient by DP
Binomial coefficients are coefficients of the binomial formula: (a + b)n = C(n,0)anb C(n,k)an-kbk C(n,n)a0bn C(n, k), the number of combinations of k elements from an n-element set (0  k  n) Recurrence: C(n,k) = ? those without last element – the number of such combinations is ? those with the last element -- the number of such combinations is ?

Algorithm for Recursive Definition
function C(n,k) if k = 0 or k = n then return 1 else return C(n-1, k-1) + C(n-1, k)

C(5,3) C(4,2) C(4,3) C(3,1) C(3,2) C(3,2) C(3,3)

Time complexity is (n!) But we did a lot of duplicate computation
Dynamic programming approach Store solutions to sub-problems in a table Bottom-up approach

Computing a binomial coefficient by DP
Binomial coefficients are coefficients of the binomial formula: (a + b)n = C(n,0)anb C(n,k)an-kbk C(n,n)a0bn Recurrence: C(n,k) = C(n-1,k) + C(n-1,k-1) for n > k > 0 C(n,0) = 1, C(n,n) = 1 for n  0 Value of C(n,k) can be computed by filling a table: k k 0 1 . n C(n-1,k-1) C(n-1,k) n C(n,k)

Computing C(n,k): pseudocode and analysis
Time efficiency: Θ(nk) Space efficiency: Θ(nk)

Exercise Compute C(6, 3) by applying the dynamic programming algorithm

Longest Common Subsequence (LCS)
A subsequence of a sequence/string S is obtained by deleting zero or more symbols from S. For example, the following are some subsequences of “president”: pred, sdn, predent. In other words, the letters of a subsequence of S appear in order in S, but they are not required to be consecutive. The longest common subsequence problem is to find a maximum length common subsequence between two sequences.

LCS Sequence 1: president Sequence 2: providence Its LCS is priden.
For instance, Sequence 1: president Sequence 2: providence Its LCS is priden. president providence

LCS Sequence 1: algorithm Sequence 2: alignment
Another example: Sequence 1: algorithm Sequence 2: alignment One of its LCS is algm. a l g o r i t h m a l i g n m e n t

How to compute LCS? Let A=a1a2…am and B=b1b2…bn .
len(i, j): the length of an LCS between a1a2…ai and b1b2…bj With proper initializations, len(i, j) can be computed as follows

Running time and memory: O(mn) and O(mn).

The backtracing algorithm

Knapsack problem Given some items, pack the knapsack to get
the maximum total value. Each item has some weight and some value. Total weight that we can carry is no more than some fixed number W. So we must consider weights of items as well as their values. Item # Weight Value

Knapsack problem There are two versions of the problem:
Items are indivisible; you either take an item or not. Some special instances can be solved with dynamic programming “Fractional knapsack problem” Items are divisible: you can take any fraction of an item

0-1 Knapsack problem Given a knapsack with maximum capacity W, and a set S consisting of n items Each item i has some weight wi and benefit value bi (all wi and W are integer values) Problem: How to pack the knapsack to achieve maximum total value of packed items?

0-1 Knapsack problem Problem, in other words, is to find
The problem is called a “0-1” problem, because each item must be entirely accepted or rejected.

0-1 Knapsack problem: brute-force approach
Let’s first solve this problem with a straightforward algorithm Since there are n items, there are 2n possible combinations of items. We go through all combinations and find the one with maximum value and with total weight less or equal to W Running time will be O(2n)

0-1 Knapsack problem: dynamic programming approach
We can do better with an algorithm based on dynamic programming We need to carefully identify the subproblems

Defining a Subproblem Given a knapsack with maximum capacity W, and a set S consisting of n items Each item i has some weight wi and benefit value bi (all wi and W are integer values) Problem: How to pack the knapsack to achieve maximum total value of packed items?

Defining a Subproblem We can do better with an algorithm based on dynamic programming We need to carefully identify the subproblems Let’s try this: If items are labeled 1..n, then a subproblem would be to find an optimal solution for Sk = {items labeled 1, 2, .. k}

Defining a Subproblem If items are labeled 1..n, then a subproblem would be to find an optimal solution for Sk = {items labeled 1, 2, .. k} This is a reasonable subproblem definition. The question is: can we describe the final solution (Sn ) in terms of subproblems (Sk)? Unfortunately, we can’t do that.

? Defining a Subproblem wi bi
Weight Benefit wi bi Item # ? 1 2 3 Max weight: W = 20 For S4: Total weight: 14 Maximum benefit: 20 S4 2 4 5 S5 3 5 8 4 3 4 5 9 10 w1 =2 b1 =3 w2 =4 b2 =5 w3 =5 b3 =8 w5 =9 b5 =10 Solution for S4 is not part of the solution for S5!!! For S5: Total weight: 20 Maximum benefit: 26

Defining a Subproblem As we have seen, the solution for S4 is not part of the solution for S5 So our definition of a subproblem is flawed and we need another one!

Defining a Subproblem Given a knapsack with maximum capacity W, and a set S consisting of n items Each item i has some weight wi and benefit value bi (all wi and W are integer values) Problem: How to pack the knapsack to achieve maximum total value of packed items?

Defining a Subproblem Let’s add another parameter: w, which will represent the maximum weight for each subset of items The subproblem then will be to compute V[k,w], i.e., to find an optimal solution for Sk = {items labeled 1, 2, .. k} in a knapsack of size w

Recursive Formula for subproblems
The subproblem will then be to compute V[k,w], i.e., to find an optimal solution for Sk = {items labeled 1, 2, .. k} in a knapsack of size w Assuming knowing V[i, j], where i=0,1, 2, … k-1, j=0,1,2, …w, how to derive V[k,w]?

Recursive Formula for subproblems (continued)
It means, that the best subset of Sk that has total weight w is: 1) the best subset of Sk-1 that has total weight  w, or 2) the best subset of Sk-1 that has total weight  w-wk plus the item k

Recursive Formula The best subset of Sk that has the total weight  w, either contains item k or not. First case: wk>w. Item k can’t be part of the solution, since if it was, the total weight would be > w, which is unacceptable. Second case: wk  w. Then the item k can be in the solution, and we choose the case with greater value.

0-1 Knapsack Algorithm for w = 0 to W V[0,w] = 0 for i = 1 to n
V[i,0] = 0 if wi <= w // item i can be part of the solution if bi + V[i-1,w-wi] > V[i-1,w] V[i,w] = bi + V[i-1,w- wi] else V[i,w] = V[i-1,w] else V[i,w] = V[i-1,w] // wi > w

Remember that the brute-force algorithm
Running time for w = 0 to W V[0,w] = 0 for i = 1 to n V[i,0] = 0 < the rest of the code > O(W) Repeat n times O(W) What is the running time of this algorithm? O(n*W) Remember that the brute-force algorithm takes O(2n)

Example Let’s run our algorithm on the following data:
n = 4 (# of elements) W = 5 (max weight) Elements (weight, benefit): (2,3), (3,4), (4,5), (5,6)

Example (2) i\W 1 2 3 4 5 1 2 3 4 for w = 0 to W V[0,w] = 0

Example (3) 1 2 3 4 5 i\W for i = 1 to n V[i,0] = 0

Example (4) Items: 1: (2,3) 2: (3,4) 3: (4,5) 4: (5,6) i=1 bi=3 wi=2
1 2 3 4 5 i=1 bi=3 wi=2 w=1 w-wi =-1 1 2 3 4 if wi <= w // item i can be part of the solution if bi + V[i-1,w-wi] > V[i-1,w] V[i,w] = bi + V[i-1,w- wi] else V[i,w] = V[i-1,w] else V[i,w] = V[i-1,w] // wi > w

1 2 3 4 5 i\W i=1 bi=3 wi=2 w=2 w-wi =0 3 if wi <= w // item i can be part of the solution if bi + V[i-1,w-wi] > V[i-1,w] V[i,w] = bi + V[i-1,w- wi] else V[i,w] = V[i-1,w] else V[i,w] = V[i-1,w] // wi > w

1 2 3 4 5 i\W i=1 bi=3 wi=2 w=3 w-wi =1 3 3 if wi <= w // item i can be part of the solution if bi + V[i-1,w-wi] > V[i-1,w] V[i,w] = bi + V[i-1,w- wi] else V[i,w] = V[i-1,w] else V[i,w] = V[i-1,w] // wi > w

1 2 3 4 5 i\W i=1 bi=3 wi=2 w=4 w-wi =2 3 3 3 if wi <= w // item i can be part of the solution if bi + V[i-1,w-wi] > V[i-1,w] V[i,w] = bi + V[i-1,w- wi] else V[i,w] = V[i-1,w] else V[i,w] = V[i-1,w] // wi > w

1 2 3 4 5 i\W i=1 bi=3 wi=2 w=5 w-wi =3 3 3 3 3 if wi <= w // item i can be part of the solution if bi + V[i-1,w-wi] > V[i-1,w] V[i,w] = bi + V[i-1,w- wi] else V[i,w] = V[i-1,w] else V[i,w] = V[i-1,w] // wi > w

1 2 3 4 5 i\W i=2 bi=4 wi=3 w=1 w-wi =-2 3 3 3 3 if wi <= w // item i can be part of the solution if bi + V[i-1,w-wi] > V[i-1,w] V[i,w] = bi + V[i-1,w- wi] else V[i,w] = V[i-1,w] else V[i,w] = V[i-1,w] // wi > w

1 2 3 4 5 i\W i=2 bi=4 wi=3 w=2 w-wi =-1 3 3 3 3 3 if wi <= w // item i can be part of the solution if bi + V[i-1,w-wi] > V[i-1,w] V[i,w] = bi + V[i-1,w- wi] else V[i,w] = V[i-1,w] else V[i,w] = V[i-1,w] // wi > w

1 2 3 4 5 i\W i=2 bi=4 wi=3 w=3 w-wi =0 3 3 3 3 3 4 if wi <= w // item i can be part of the solution if bi + V[i-1,w-wi] > V[i-1,w] V[i,w] = bi + V[i-1,w- w i] else V[i,w] = V[i-1,w] else V[i,w] = V[i-1,w] // wi > w

1 2 3 4 5 i\W i=2 bi=4 wi=3 w=4 w-wi =1 3 3 3 3 3 4 4 if wi <= w // item i can be part of the solution if bi + V[i-1,w-wi] > V[i-1,w] V[i,w] = bi + V[i-1,w- wi] else V[i,w] = V[i-1,w] else V[i,w] = V[i-1,w] // wi > w

1 2 3 4 5 i\W i=2 bi=4 wi=3 w=5 w-wi =2 3 3 3 3 3 4 4 7 if wi <= w // item i can be part of the solution if bi + V[i-1,w-wi] > V[i-1,w] V[i,w] = bi + V[i-1,w- wi] else V[i,w] = V[i-1,w] else V[i,w] = V[i-1,w] // wi > w

1 2 3 4 5 i\W i=3 bi=5 wi=4 w= 1..3 3 3 3 3 3 4 4 7 3 4 if wi <= w // item i can be part of the solution if bi + V[i-1,w-wi] > V[i-1,w] V[i,w] = bi + V[i-1,w- wi] else V[i,w] = V[i-1,w] else V[i,w] = V[i-1,w] // wi > w

1 2 3 4 5 i\W i=3 bi=5 wi=4 w= 4 w- wi=0 3 3 3 3 3 4 4 7 3 4 5 if wi <= w // item i can be part of the solution if bi + V[i-1,w-wi] > V[i-1,w] V[i,w] = bi + V[i-1,w- wi] else V[i,w] = V[i-1,w] else V[i,w] = V[i-1,w] // wi > w

1 2 3 4 5 i\W i=3 bi=5 wi=4 w= 5 w- wi=1 3 3 3 3 3 4 4 7 3 4 5 7 if wi <= w // item i can be part of the solution if bi + V[i-1,w-wi] > V[i-1,w] V[i,w] = bi + V[i-1,w- wi] else V[i,w] = V[i-1,w] else V[i,w] = V[i-1,w] // wi > w

1 2 3 4 5 i\W i=4 bi=6 wi=5 w= 1..4 3 3 3 3 3 4 4 7 3 4 5 7 3 4 5 if wi <= w // item i can be part of the solution if bi + V[i-1,w-wi] > V[i-1,w] V[i,w] = bi + V[i-1,w- wi] else V[i,w] = V[i-1,w] else V[i,w] = V[i-1,w] // wi > w

1 2 3 4 5 i\W i=4 bi=6 wi=5 w= 5 w- wi=0 3 3 3 3 3 4 4 7 3 4 5 7 3 4 5 7 if wi <= w // item i can be part of the solution if bi + V[i-1,w-wi] > V[i-1,w] V[i,w] = bi + V[i-1,w- wi] else V[i,w] = V[i-1,w] else V[i,w] = V[i-1,w] // wi > w

Comments This algorithm only finds the max possible value that can be carried in the knapsack i.e., the value in V[n,W] To know the items that make this maximum value, an addition to this algorithm is necessary

How to find actual Knapsack Items
All of the information we need is in the table. V[n,W] is the maximal value of items that can be placed in the Knapsack. Let i=n and k=W if V[i,k]  V[i1,k] then mark the ith item as in the knapsack i = i1, k = k-wi else i = i1 // Assume the ith item is not in the knapsack // Could it be in the optimally packed knapsack?

Finding the Items Items: 1: (2,3) 2: (3,4) 3: (4,5) 4: (5,6) 1 2 3 4 5
1 2 3 4 5 i\W i=4 k= 5 bi=6 wi=5 V[i,k] = 7 V[i1,k] =7 3 3 3 3 3 4 4 7 3 4 5 7 3 4 5 7 i=n, k=W while i,k > 0 if V[i,k]  V[i1,k] then mark the ith item as in the knapsack i = i1, k = k-wi else i = i1

Finding the Items (2) Items: 1: (2,3) 2: (3,4) 3: (4,5) 4: (5,6) 1 2 3

1 2 3 4 5 i\W i=2 k= 5 bi=4 wi=3 V[i,k] = 7 V[i1,k] =3 k  wi=2 3 3 3 3 3 4 4 7 7 3 4 5 7 3 4 5 7 i=n, k=W while i,k > 0 if V[i,k]  V[i1,k] then mark the ith item as in the knapsack i = i1, k = k-wi else i = i1

1 2 3 4 5 i\W i=1 k= 2 bi=3 wi=2 V[i,k] = 3 V[i1,k] =0 k  wi=0 3 3 3 3 3 3 4 4 7 3 4 5 7 3 4 5 7 i=n, k=W while i,k > 0 if V[i,k]  V[i1,k] then mark the ith item as in the knapsack i = i1, k = k-wi else i = i1

1 2 3 4 5 i\W i=0 k= 0 3 3 3 3 3 4 4 7 The optimal knapsack should contain {1, 2} 3 4 5 7 3 4 5 7 i=n, k=W while i,k > 0 if V[i,k]  V[i1,k] then mark the nth item as in the knapsack i = i1, k = k-wi else i = i1

1 2 3 4 5 i\W 3 3 3 3 3 3 4 4 7 7 The optimal knapsack should contain {1, 2} 3 4 5 7 3 4 5 7 i=n, k=W while i,k > 0 if V[i,k]  V[i1,k] then mark the nth item as in the knapsack i = i1, k = k-wi else i = i1

Dynamic Programming General approach – combine solutions to subproblems to get the solution to an problem Unlike Divide and Conquer in that subproblems are dependent rather than independent bottom-up approach save values of subproblems in a table and use more than one time

Matrix Chain Multiplication
A1 A2 A3 An Matrix multiplication is associative All ways that a sequence can be parenthesized give the same answer But some are much less expensive to compute

Matrix Chain Multiplication Problem
Given a chain < A1,A2, . . .,An> of n matrices, where i = 1, 2, . . ., n and matrix Ai has dimension pi-1 pi, fully parenthesize the product A1 A2 An in a way that minimizes the number of scalar multiplications

Example Matrix Dimensions A 13 x 5 B 5 X 89 C 89 X 3 D 3 X 34
M = A B C D 13 x 34

Parenthesization Scalar multiplications
1 ((A B) C) D 10,582 2 (A B) (C D) 54,201 3 (A (B C)) D 2, 856 4 A ((B C) D) 4, 055 5 A (B (C D)) 26,418 x 5 x 89 to get (A B) 13 x 89 result 13 x 89 x 3 to get ((AB)C) 13 x 3 result 13 x 3 x 34 to get (((AB)C)D) 13 x 34

Number of Parenthesizations

T(n) ways to parenthesize

Define value of the optimal solution recursively in terms of optimal solutions to sub-problems.
Consider the problem Ai..j Let m[i,j] be the minimum number of scalar multiplications needed to compute matrix Ai..j The cost of the cheapest way to compute A1..n is m[1,n]

Define m[i,j] If i = j the chain has one matrix and the cost m[i,i] = 0 for all i  n If i < j Assume the optimal split is at position k where i k < j m[i,j] = m[i,k] + m[k+1,j] + pi-1 pkpj to compute Ai..k and Ak+1..j

Problem: we don’t know what the value for k is, but there are only j-i possibilities

Compute optimal cost by using a bottom-up approach
Example: 4 matrix chain Matrix Dimensions A x 1 A x 100 A x 1 A x 100 po p p p p4

j j i i 1 2 3 4 1 2 3 4 s m po p p p p4

MATRIX-CHAIN-ORDER(p)
1 n length[p] - 1 2 for i 1 to n do m[i,i] 0 4 for l 2 to n do for i 1 to n - l + 1 do ji + l - 1 m[i,j]  for k i to j - 1 do q m[i,k]+m[k+1,j]+ pi-1 pkpj if q < m[i,j] then m[i,j] = q s[i,j] = k 13 return m and s

Time and Space Complexity
Time complexity Triply nested loops O(n3) Space complexity n x n two dimensional table O(n2)

Matrix-Chain-Order determines the optimal number of scalar multiplications
Does not directly compute the product The s matrix contains the optimal split for every level

MATRIX-CHAIN-MULTIPLY(A,s,i,j)
1 if j > i 2 then X MATRIX-CHAIN-MULTIPLY(A,s,i,s[i,j]) Y MATRIX-CHAIN-MULTIPLY(A,s,s[i,j]+1,j) return MATRIX-MULTIPLY(X,Y) 5 else return Ai Initial call MATRIX-CHAIN-MULTIPLY(A,s,1,n) where A = < A1,A2, . . .,An>

Example Show how to multiply this matrix chain optimally
Solution on the board Minimum cost 15,125 Optimal parenthesization ((A1(A2A3))((A4 A5)A6)) Matrix Dimension A1 30×35 A2 35×15 A3 15×5 A4 5×10 A5 10×20 A6 20×25

Optimal Polygon Triangulation
A triangulation of a polygon is a set of T chords of the polygon that divide the polygon into disjoint triangles. In a triangulation, no chords intersect (except at end-points) . The sides of triangles produced by the triangulation are either chords in the triangulation or sides of the polygon. Every triangulation of an n-vertex convex polygon has n-3 chords and divides the polygon into n-2 triangles. a b.

Labeling a convex polygon
v0= vn v4 v1 v3 P = <v0, v1, vn-1> v2

Chords v5 v0 v5 v0 v4 v4 v1 v1 v3 v3 v2 v2

Optimal Polygon Triangulation cont.
In the optimal polygon triangulation problem, we are given a polygon P = {v0, v1, v2, …., vn-1} and a weight function w defined on triangles formed by sides and chords of P. The problem is to find a triangulation that minimizes the sum of the weights of the triangles in the triangulation. This problem, like matrix chain multiplication, uses parenthesization. A full parenthesization corresponds to a full binary tree also called a parse tree.

Parenthesization in Triangulation
Above is the parse tree for the triangulation of a polygon.The internal nodes of the parse tree are the chords of the triangulation plus the side v0v6, which is the root.

((A1(A2A3)(A4(A5A6))) A1(A2A3) A4(A5A6) A1 A2A3 A4 A5A6 A2 A3 A5 A6

((A1(A2A3)(A4(A5A6))) v0 v6 v1 v5 v2 v4 v3

Optimal Polygon Triangulation
Mapping one problem to another kind of problem for which you already have a good solution is a useful concept

Optimal (polygon) triangulation problem
Given a convex polygon P = <v0, v1, vn-1> and a weight function w defined on triangles formed by sides and chords of P. Find a triangulation that minimizes the sum of the weights of the triangles in the triangulation One possible weight function is to minimize the sum of the lengths of the sides

Triangulation and Matrix Chain Multiplication
Since a fully parenthesized product of n matrices corresponds to a parse tree with n leaves, it therefore also corresponds to a triangulation of an (n+1)-vertex polygon. Each matrix Ai in a product of A1A2…An corresponds to side vi-1vi of an (n+1)-vertex polygon.

Triangulation and Matrix Chain Multiplication cont.
Given a matrix chain product A1A2….AN, we define an (n+1)-vertex convex polygon p = {v0,v1,….,vn}. If matrix Ai has dimensions pi-1 x pi, for I = 1,2,…,n, the weight function for the triangulation is defined as w(Δvivjvk) = pipjpk An optimal triangulation of P with respect to this weight function gives the parse tree for an optimal parenthesization of A1A2….An.

Triangulation A triangulation of a polygon is a set T of chords of the polygon that divide the polygon into disjoint triangles. No chords intersect The set of chords T is maximal; every chord not in T intersects some chord in T Every triangulation of an n-vertex convex polygon has n-3 chords divides polygon into n-2 triangles

Correspondance of Parenthesization
We will show how to view both parenthesization and triangulation as parse trees and show the correspondence between the two views. The result will be that a slight modification of the matrix chain problem can be used to solve the triangulation problem.

Greedy Algorithms Main Concept: Make the best or greedy choice at any given step. Choices are made in sequence such that Each individual choice is best according to some limited “short-term” criterion, that is not too expensive to evaluate Once a choice is made, it cannot be undone! Even if it becomes evident later that it was a poor choice The goal is to take a sequence of locally optimal actions, hoping to yield a globally optimal solution.

Obstacle avoiding shortest A-B path
shortest path greedy choice A B

When to be Greedy Greedy algorithms apply to problems with
Optimal substructures: optimal solutions contain optimal sub-solutions. The greedy choice property: an optimal solution can be obtained by making the greedy choice at each step. Many optimal solutions, but we only need one such solution.

COIN CHANGE MAKING Use minimum number of coins to make change for a given amount. Greedy: pick the largest coin that fits first, then iterate.

The Coin Change Making Problem
PROBLEM: Within a coin denomination system, make change for a given amount S, using the fewest number of coins possible. Greedy Strategy (the obvious one): Commit to the largest coin denomination that fits within the (remaining) amount, then iterate. Greedy(S) = the greedy solution for amount S. Optimum(S) = the optimum solution for amount S. Example 1: coin system: 25, 10, 5, 1. S = (Two of many feasible solutions are shown below.) = coins used = Optimum solution uses 8 coins. Greedy(98) = Optimum(98). Example 2: Imaginary coin denomination system: 7, 5, 1 kraaps. S = 10 = (Optimum) = (Greedy). Greedy(10) ≠Optimum(10) in this system.

The Problem Formulation
INPUT: Coin denomination system a = a1 , a2 , … , an , a1 > a2 > … > an = 1, and S (all positive integers). OUTPUT: The solution x =  x1 , x2 , … , xn , where xt = the number of coin type at used, for t = 1..n. FEAS: a1x1 + a2x … + anxn = S, & xt is a non-negative integer, for t=1..n. GOAL: Minimize objective cost x1 + x2 + … + xn. We need an =1 to have a feasible soln for every S. minimize x1 + x2 + ··· + xn subject to: (1) a1x1 + a2x2 + ··· + anxn = S (2) xtN, for t = 1..n objective function feasibility constraints

Efficient implementation
minimize x1 + x2 + ··· + xn subject to: (1) a1x1 + a2x2 + ··· + anxn = S (2) xtN, for t = 1..n objective function feasibility constraints Algorithm Greedy( a1 , a2 , … , an , S) § takes O(n) time CoinCount  0; U  S for t  1 .. n do § largest coin denomination first xt  U div at § xt  U/at U  U mod at § U  U – at xt CoinCount  CoinCount + xt § objective value end-for return ( x1 , x2 , … , xn ; CoinCount) end

Graph Graph G = (V, E) V = V(G) = vertex set of G E = E(G) = edge set of G ( a set of pairs of vertices of G) 1 2 4 5 3 6 V = { 1, 2, 3, 4, 5, 6 } E = {(1,2), (1,4), (1,5), (2,3), (2,5), (3,5), (4,5)} Undirected graph: edges are unordered pairs of vertices: Directed graph (or digraph): edges are ordered pairs of vertices: V = { 1, 2, 3, 4, 5, 6 } E = { (1,2), (1,4), (1,5), (3,2), (4,4), (4,5), (5,1), (5,2), (5,3)} 1 2 4 5 3 6

Edge Weighted Graph G = (V, E, w) w: E |  1 2 4 5 3 6
e.g., w(1,5) = w(5,1) = 17. 13 -16 14 17 23 21 18 1 2 4 5 3 6 E = { (1,2,13), (1,4,14), (1,5,15), (3,2,-16) (4,5,23), (5,1,14), (5,2,21), (5,3,18)} e.g., w(1,5) = 15, w(5,1) = 14. 13 -16 14 15 23 21 18

Adjacency Matrix 1 2 3 4 5 6 A = 1 2 3 4 5 6 A =

Weighted Adjacency Matrix
 4 -6 5 2 9 8 6 1 3 A =  -6 5 7 4 8 3 9 6 1 2 A =

(Weighted) Adjacency List Structure
1 2 3 4 5 6 9 8 -6 Adj 1 2 3 4 5 6 9 8 7 -6 Adj

The Hand-Shaking Lemma
Vertex vV(G): degree (or valance) , in-degree, out-degree Undirected G: deg(v) = | { u | (v,u)  E(G) } | = | Adj[v] | Digraph G: outdeg(v) = | { u | (v,u)  E(G) } | = | Adj[v] | indeg(v) = | { u | (u,v)  E(G) } | deg(v) = outdeg(v) + indeg(v) The Hand-Shaking Lemma: For any graph (directed or undirected) we have: For any directed graph we also have:

Adjacency Matrix vs Adjacency List Structure
complexity Adjacency Matrix Adjacency List # memory cells Space O(V2) O(V + E) Initialize structure Time Scan (incident edges of) all vertices List vertices adjacent to uV(G) O(V) O(|Adj[u]|) Is (u,v)  E(G)? O(1)

Minimum Spanning Tree (MST)
A spanning tree for a connected, undirected graph, G=(V, E) is a connected subgraph of G that forms an undirected tree incident with each vertex. In a weighted graph G=(V, E, W), the weight of a subgraph is the sum of the weights of the edges in the subgraph. A minimum spanning tree (MST) for a weighted graph is a spanning tree with the minimum weight.

Another spanning tree T’ Cost(T’) = 3+5+4+8 = 20 a
b e d c 5 3 8 4 Another spanning tree T’ Cost(T’) = = 20 a b e d c 3 2 4 7 Minimum spanning tree T Cost(T) = = 16 a b e d c 5 3 2 8 4 7 G

Minimum Spanning Tree Problem: given a connected, undirected, weighted graph: find a spanning tree using edges that minimize the total weight 6 4 5 9 14 2 10 15 3 8

Minimum Spanning Tree Which edges form the minimum spanning tree (MST) of the graph? A 6 4 5 9 H B C 14 2 10 15 G E D 3 8 F

Minimum Spanning Tree Answer: A 6 4 5 9 H B C 14 2 10 15 G E D 3 8 F

Another Example Given a weighted graph G=(V, E,W), find a MST of G 7 3
6 4 5 3 5 2 6 3

Finding a MST MSTs satisfy the optimal substructure property: an optimal tree is composed of optimal subtrees Principal greedy methods: algorithms by Prim and Kruskal Prim Grow a single tree by repeatedly adding the least cost edge that connects a vertex in the existing tree to a vertex not in the existing tree Intermediary solution is a subtree Kruskal Grow a tree by repeatedly adding the least cost edge that does not introduce a cycle among the edges included so far Intermediary solution is a spanning forest

Prime’s Algorithm (High-Level Pseudocode)
Prime(G) //Input: A weighted connected graph G = <V, E> //Output: ET --- the set of edges composing MST of G VT = {v0} ET =  for i = 1 to |V| - 1 do find a minimum-weight edge e* = (u*, v*) among all the edges (u, v) such that u is in VT and v is in V-VT VT = VT  {v*} ET = ET  {e*} return ET

Prime’s Algorithm (High-Level Pseudocode)
Prime(G) //Input: A weighted connected graph G = <V, E> //Output: ET --- the set of edges composing MST of G VT = {v0} ET =  for i = 1 to |V| - 1 do find a minimum-weight edge e* = (u*, v*) among all the edges (u, v) such that u is in VT and v is in V-VT VT = VT  {v*} ET = ET  {e*} return ET A 6 4 9 5 H B C 14 2 10 15 G E D 3 8 F

Prim’s Algorithm Grow a single tree by repeatedly adding the least cost edge that connects a vertex in the existing tree to a vertex not in the existing tree Intermediary solution is a subtree MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v);

Prim’s Algorithm MST-Prim(G, w, r) Q = V[G]; for each u  Q
6 4 MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 9 5 14 2 10 15 3 8 Run on example graph

Prim’s Algorithm  MST-Prim(G, w, r) Q = V[G]; for each u  Q
6 4 MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 9 5    14 2 10 15    3 8  Run on example graph

6 4 MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 9 5    14 2 10 15 r   3 8  Pick a start vertex r

Red vertices have been removed from Q
Prim’s Algorithm  6 4 MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 9 5    14 2 10 15 u   3 8  Red vertices have been removed from Q

Red arrows indicate parent pointers
Prim’s Algorithm  6 4 MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 9 5    14 2 10 15 u   3 8 3 Red arrows indicate parent pointers

6 4 MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 9 5 14   14 2 10 15 u   3 8 3

6 4 MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 9 5 14   14 2 10 15   3 8 3 u

6 4 MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 9 5 14   14 2 10 15 8  3 8 3 u

6 4 MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 9 5 10   14 2 10 15 8  3 8 3 u

6 4 MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 9 5 10 2  14 2 10 15 8  3 8 3 u

6 4 MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 9 5 10 2  14 2 10 15 8 15 3 8 3 u

Prim’s Algorithm u  MST-Prim(G, w, r) Q = V[G]; for each u  Q
6 4 MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 9 5 10 2  14 2 10 15 8 15 3 8 3

Prim’s Algorithm u  MST-Prim(G, w, r) Q = V[G]; for each u  Q
6 4 MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 9 5 10 2 9 14 2 10 15 8 15 3 8 3

Prim’s Algorithm u 4 MST-Prim(G, w, r) Q = V[G]; for each u  Q

Prim’s Algorithm 4 MST-Prim(G, w, r) Q = V[G]; for each u  Q
6 4 MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); 9 5 5 2 9 14 2 u 10 15 8 15 3 8 3

Review: Prim’s Algorithm
MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); What is the hidden cost in this code?

MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; DecreaseKey(v, w(u,v)); delete the smallest element from the min-heap decrease an element’s value in the min-heap (outline an efficient algorithm for it)

MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; DecreaseKey(v, w(u,v)); How often is ExtractMin() called? How often is DecreaseKey() called?

What will be the running time? MST-Prim(G, w, r) Q = V[G]; for each u  Q key[u] = ; key[r] = 0; p[r] = NULL; while (Q not empty) u = ExtractMin(Q); for each v  Adj[u] if (v  Q and w(u,v) < key[v]) p[v] = u; key[v] = w(u,v); There are n=|V| ExtractMin calls and m=|E| DecreaseKey calls. The priority Q implementation has a large impact on performance. E.g., O((n+m)lg n) = O(m lg n) using min-heap for Q (for connected graph, n – 1  m)

Finding a MST Principal greedy methods: algorithms by Prim and Kruskal
Grow a single tree by repeatedly adding the least cost edge that connects a vertex in the existing tree to a vertex not in the existing tree Intermediary solution is a subtree Kruskal Grow a tree by repeatedly adding the least cost edge that does not introduce a cycle among the edges included so far Intermediary solution is a spanning forest

MST Applications? A 6 4 5 9 H B C 14 2 10 15 G E D 3 8 F

MST Applications Network design
telephone, electrical power, hydraulic, TV cable, computer, road MST gives a minimum-cost network connecting all sites MST: the most economical construction of a network

Computational Complexity Classes
Computable Uncomputable EXP = exponential P = polynomial

P and NP Tractability Some problems are intractable: as they grow large, we are unable to solve them in reasonable time What constitutes reasonable time? Standard working definition: polynomial time On an input of size n the worst-case running time is O(nk) for some constant k O(n2), O(n3), O(1), O(n lg n), O(2n), O(nn), O(n!) Polynomial time: O(n2), O(n3), O(1), O(n lg n) Not in polynomial time: O(2n), O(nn), O(n!)

P, NP, EXP “P vs NP”  “polynomial time Construction vs Verification”

Polynomial-Time Algorithms
Are some problems solvable in polynomial time? Of course: many algorithms we’ve studied provide polynomial-time solutions to some problems Are all problems solvable in polynomial time? No: Turing’s “Halting Problem” is not solvable by any computer, no matter how much time is given Most problems that do not yield polynomial-time algorithms are either optimization or decision problems.

Optimization/Decision Problems
Optimization Problems An optimization problem is one which asks, “What is the optimal solution to problem X?” Examples: 0-1 Knapsack Fractional Knapsack Minimum Spanning Tree Decision Problems An decision problem is one with yes/no answer Does a graph G have a MST of weight  W?

Optimization/Decision Problems
An optimization problem tries to find an optimal solution A decision problem tries to answer a yes/no question Many problems will have decision and optimization versions Eg: Traveling salesman problem optimization: find hamiltonian cycle of minimum weight decision: is there a hamiltonian cycle of weight  k Some problems are decidable, but intractable: as they grow large, we are unable to solve them in reasonable time Is there a polynomial-time algorithm that solves the problem? A Hamiltonian cycle (or Hamiltonian circuit) is a cycle in an undirected graph that visits each vertex exactly once and also returns to the starting vertex.

The Class P P: the class of decision problems that have polynomial-time deterministic algorithms. That is, they are solvable in O(p(n)), where p(n) is a polynomial on n A deterministic algorithm is (essentially) one that always computes the correct answer Why polynomial? if not, very inefficient nice closure properties the sum and composition of two polynomials are always polynomials too

Sample Problems in P Fractional Knapsack Sorting Others?

The class NP NP: the class of decision problems that are solvable in polynomial time on a nondeterministic machine (or with a nondeterministic algorithm) (A determinstic computer is what we know) A nondeterministic computer is one that can “guess” the right answer or solution Think of a nondeterministic computer as a parallel machine that can freely spawn an infinite number of processes Thus NP can also be thought of as the class of problems whose solutions can be verified in polynomial time Note that NP stands for “Nondeterministic Polynomial-time”

Sample Problems in NP Fractional Knapsack Sorting Others?
Hamiltonian Cycle (Traveling Salesman) Graph Coloring Satisfiability (SAT) the problem of deciding whether a given Boolean formula is satisfiable

The Satisfiability (SAT) Problem
Given a Boolean expression on n variables, can we assign values such that the expression is TRUE? Seems simple enough, but no known deterministic polynomial time algorithm exists Easy to verify in polynomial time!

Example: CNF satisfiability
This problem is in NP. Nondeterministic algorithm: Guess truth assignment Check assignment to see if it satisfies CNF formula Example: (A¬B  ¬C )  (¬A  B)  (¬ B  D  F )  (F  ¬ D) Truth assignments: A B C D E F ... (how many more?) Checking phase: Θ(n) conjunctive normal form (CNF)

Review: P And NP Summary
P = set of problems that can be solved in polynomial time Examples: Fractional Knapsack, … NP = set of problems for which a solution can be verified in polynomial time Examples: Fractional Knapsack,…, Hamiltonian Cycle, CNF SAT, 3-CNF SAT Clearly P  NP Open question: Does P = NP? Most suspect not An August 2010 claim of proof that P ≠ NP, by Vinay Deolalikar, researcher at HP Labs, Palo Alto, has flaws a 3-CNF formula (i.e. with at most 3 variables per clause)

Analysis and Design of Algorithms

Similar presentations

Presentation on theme: "Analysis and Design of Algorithms"— Presentation transcript:

Similar presentations

About project

Feedback

Log in

Auth with social network:

Analysis and Design of Algorithms

Similar presentations

Presentation on theme: "Analysis and Design of Algorithms"— Presentation transcript:

Similar presentations

About project

Feedback