1. CS 3343: Analysis of Algorithms
Review for Exam 2

2. Exam 2
Closed book exam
One cheat sheet allowed (limit to a single page of letter-size paper, double-sided)
Thursday, April 18, class time + 5 minutes
Basic calculator (no graphing) is allowed but not necessary

3. Materials covered Quick Sort Heap sort, priority queue
Linear time sorting algorithms
Order statistics
Dynamic programming
Greedy algorithm
Questions will be similar to homework / quizzes
Familiar with the algorithm procedure
Some analysis of time/space complexity
One or two problems on algorithm design

4. QuickSort Quicksort algorithm Randimized Quicksort algorithm
Partition algorithm
Quicksort/Randomized Quicksort time complexity analysis
Back

5. Quick sort Quicksort an n-element array:
Divide: Partition the array into two subarrays around a pivot x such that elements in lower subarray £ x £ elements in upper subarray.
Conquer: Recursively sort the two subarrays.
Combine: Trivial.
£ x x
≥ x
Key: Linear-time partitioning subroutine.

6. Pseudocode for quicksort
QUICKSORT(A, p, r)
if p < r
then q  PARTITION(A, p, r) QUICKSORT(A, p, q–1)
QUICKSORT(A, q+1, r)
Initial call: QUICKSORT(A, 1, n)

7. Partition Code Partition(A, p, r)
x = A[p]; // pivot is the first element
i = p;
j = r + 1;
while (TRUE) {
repeat
i++;
until A[i] > x | i >= j;
j--;
until A[j] < x | j <= i;
if (i < j)
Swap (A[i], A[j]);
else
break; }
swap (A[p], A[j]);
return j;

8. p r 6 10 5 8 13 3 2 11
x = 6 i j 6 10 5 8 13 3 2 11
scan i j 6 2 5 8 13 3 10 11
swap i j
Partition
example 6 2 5 8 13 3 10 11
scan i j 6 2 5 3 13 8 10 11
swap i j 6 2 5 3 13 8 10 11
scan j i p q r 3 2 5 6 13 8 10 11
final swap

9. 6 10 5 8 11 3 2 13
Quick sort
example 3 2 5 6 11 8 10 13 2 3 5 6 10 8 11 13 2 3 5 6 8 10 11 13 2 3 5 6 8 10 11 13

10. Quicksort Runtimes Best case runtime Tbest(n)  O(n log n)
Worst case runtime Tworst(n)  O(n2)
Average case runtime Tavg(n)  O(n log n)
Expected runtime of randomized quicksort is O(n log n)

11. Randomized Partition Randomly choose an element as pivot
Every time need to do a partition, throw a die to decide which element to use as the pivot
Each element has 1/n probability to be selected
Rand-Partition(A, p, r){
d = random(); // draw a random number between 0 and 1
index = p + floor((r-p+1) * d); // p<=index<=q
swap(A[p], A[index]);
Partition(A, p, r); // now use A[p] as pivot }

12. Running time of randomized quicksort
T(0) + T(n–1) + dn if 0 : n–1 split,
T(1) + T(n–2) + dn if 1 : n–2 split, M
T(n–1) + T(0) + dn if n–1 : 0 split,
T(n) =
The expected running time is an average of all cases
Expectation

13. Need to Prove: T(n) ≤ c n log (n)
Fact:
Need to Prove: T(n) ≤ c n log (n)
Assumption: T(k) ≤ ck log (k) for 0 ≤ k ≤ n-1
Prove by induction
If c ≥ 4

14. Heaps Heap definition Heapify Buildheap Heapsort
Procedure and running time (!)
Heapsort
Comparison with other sorting algorithms in terms of running time, stability, and in-place.
Priority Queue operations
Back

15. Heaps A heap can be seen as a complete binary tree:
Perfect binary tree 16 14 10 8 7 9 3 2 4 1 16 14 10 8 7 9 3 2 4 1

16. Referencing Heap Elements
So…
Parent(i)
{return i/2;}
Left(i)
{return 2*i;}
right(i)
{return 2*i + 1;}

17. Heap Operations: Heapify()
Heapify(A, i)
{ // precondition: subtrees rooted at l and r are heaps
l = Left(i); r = Right(i);
if (l <= heap_size(A) && A[l] > A[i])
largest = l;
else
largest = i;
if (r <= heap_size(A) && A[r] > A[largest])
largest = r;
if (largest != i) {
Swap(A, i, largest);
Heapify(A, largest); }
} // postcondition: subtree rooted at i is a heap
Among A[l], A[i], A[r],
which one is largest?
If violation, fix it.

18. Heapify() Example 16 4 10 14 7 9 3 2 8 1
A = 16 4 10 14 7 9 3 2 8 1

19. Heapify() Example 16 4 10 14 7 9 3 2 8 1
A = 16 4 10 14 7 9 3 2 8 1

20. Heapify() Example 16 4 10 14 7 9 3 2 8 1
A = 16 4 10 14 7 9 3 2 8 1

21. Heapify() Example 16 14 10 4 7 9 3 2 8 1
A = 16 14 10 4 7 9 3 2 8 1

22. Heapify() Example 16 14 10 4 7 9 3 2 8 1
A = 16 14 10 4 7 9 3 2 8 1

23. Heapify() Example 16 14 10 8 7 9 3 2 4 1
A = 16 14 10 8 7 9 3 2 4 1

24. Heapify() Example 16 14 10 8 7 9 3 2 4 1
A = 16 14 10 8 7 9 3 2 4 1

25. BuildHeap() // given an unsorted array A, make A a heap BuildHeap(A) {
heap_size(A) = length(A);
for (i = length[A]/2 downto 1)
Heapify(A, i); }

26. BuildHeap() Example
Work through example A = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7} 4 1 3 2 16 9 10 14 8 7
A = 4 1 3 2 16 9 10 14 8 7

27. 4 1 3 2 16 9 10 14 8 7
A = 4 1 3 2 16 9 10 14 8 7

28. 4 1 3 2 16 9 10 14 8 7
A = 4 1 3 2 16 9 10 14 8 7

29. 4 1 3 14 16 9 10 2 8 7
A = 4 1 3 14 16 9 10 2 8 7

30. 16 14 10 8 7 9 3 2 4 1
A = 16 14 10 8 7 9 3 2 4 1

31. Analyzing BuildHeap(): Tight
To Heapify() a subtree takes O(h) time where h is the height of the subtree
h = O(lg m), m = # nodes in subtree
The height of most subtrees is small
Fact: an n-element heap has at most n/2h+1 nodes of height h
CLR 6.3 uses this fact to prove that BuildHeap() takes Θ(n) time

32. Heapsort Heapsort(A) { BuildHeap(A); for (i = length(A) downto 2)
Swap(A[1], A[i]);
heap_size(A) -= 1;
Heapify(A, 1); }

33. Heapsort Example
Work through example A = {4, 1, 3, 2, 16, 9, 10, 14, 8, 7} 4 1 3 2 16 9 10 14 8 7
A = 4 1 3 2 16 9 10 14 8 7

34. Heapsort Example First: build a heap 16 14 10 8 7 9 3 2 4 1 A = 16 14

35. Heapsort Example Swap last and first 1 14 10 8 7 9 3 2 4 16 A = 1 14

36. Heapsort Example Last element sorted 1 14 10 8 7 9 3 2 4 16 A = 1 14

37. Heapsort Example Restore heap on remaining unsorted elements 14 8 10 47 9 3 2 1 16
Heapify
A = 14 8 10 4 7 9 3 2 1 16

38. Heapsort Example Repeat: swap new last and first 1 8 10 4 7 9 3 2 1416
A = 1 8 10 4 7 9 3 2 14 16

39. Heapsort Example Restore heap 10 8 9 4 7 1 3 2 14 16 A = 10 8 9 4 7 1

40. Heapsort Example Repeat 9 8 3 4 7 1 2 10 14 16 A = 9 8 3 4 7 1 2 10 14

41. Heapsort Example Repeat 8 7 3 4 2 1 9 10 14 16 A = 8 7 3 4 2 1 9 10 14

42. Heapsort Example Repeat 1 2 3 4 7 8 9 10 14 16 A = 1 2 3 4 7 8 9 10 14

43. Implementing Priority Queues
HeapMaximum(A) {
return A[1]; }

44. Implementing Priority Queues
HeapExtractMax(A) {
if (heap_size[A] < 1) { error; }
max = A[1];
A[1] = A[heap_size[A]]
heap_size[A] --;
Heapify(A, 1);
return max; }

45. HeapExtractMax Example16 14 10 8 7 9 3 2 4 1
A = 16 14 10 8 7 9 3 2 4 1

46. HeapExtractMax Example
Swap first and last, then remove last 1 14 10 8 7 9 3 2 4 16
A = 1 14 10 8 7 9 3 2 4 16

47. HeapExtractMax Example
Heapify 14 8 10 4 7 9 3 2 1 16
A = 14 8 10 4 7 9 3 2 1 16

48. Implementing Priority Queues
HeapChangeKey(A, i, key){
if (key ≤ A[i]){ // decrease key
A[i] = key;
heapify(A, i);
} else { // increase key
while (i>1 & A[parent(i)]<A[i])
swap(A[i], A[parent(i)]; }
Sift down
Bubble up

49. HeapChangeKey Example
Increase key 16 14 10 8 7 9 3 2 4 1
A = 16 14 10 8 7 9 3 2 4 1

50. HeapChangeKey Example
Increase key 16 14 10 15 7 9 3 2 4 1
A = 16 14 10 15 7 9 3 2 4 1

51. HeapChangeKey Example
Increase key 16 15 10 14 7 9 3 2 4 1
A = 16 15 10 14 7 9 3 2 4 1

52. Implementing Priority Queues
HeapInsert(A, key) {
heap_size[A] ++;
i = heap_size[A];
A[i] = -∞;
HeapChangeKey(A, i, key); }

53. HeapInsert Example HeapInsert(A, 17) 16 14 10 8 7 9 3 2 4 1 A = 16 14

54. HeapInsert Example HeapInsert(A, 17) -∞ -∞ -∞ makes it a valid heap 1614 10 8 7 9 3 2 4 1 -∞
-∞ makes it a valid heap
A = 16 14 10 8 7 9 3 2 4 1 -∞

55. HeapInsert Example HeapInsert(A, 17) Now call changeKey 16 14 10 8 7 93 2 4 1 17
Now call changeKey
A = 16 14 10 8 7 9 3 2 4 1 17

56. HeapInsert Example HeapInsert(A, 17) 17 16 10 8 14 9 3 2 4 1 7 A = 17

57. Linear time sorting Counting sort Radix sort
Algorithm and analysis
Conditions that make counting sort linear
Stable property and memory requirement
Radix sort
Analysis
Comparison of different sorting algorithms
Running time
Memory requirement
Stable property
Back

58. Counting sort for i  1 to k do C[i]  0 for j  1 to n1.
for i  1 to k
do C[i]  0
for j  1 to n
do C[A[ j]]  C[A[ j]] + 1 ⊳ C[i] = |{key = i}|
for i  2 to k
do C[i]  C[i] + C[i–1] ⊳ C[i] = |{key £ i}|
for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
Initialize 2.
Count 3.
Compute running sum 4.
Re-arrange

59. Counting sort A: 4 1 3 4 3 C: 1 2 2 B: C': 1 1 3 5 for i  2 to k2 2 B:
C': 1 1 3 5 3.
for i  2 to k
do C[i]  C[i] + C[i–1] ⊳ C[i] = |{key £ i}|

60. Loop 4: re-arrange A: 4 1 3 4 3 C: 1 1 3 5 B: 3 C': 1 1 3 52 3 4 5 1 2 3 4 A: 4 1 3 4 3 C: 1 1 3 5 B: 3
C': 1 1 3 5 4.
for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1

61. Analysis Q(k) Q(n) Q(k) Q(n) Q(n + k) 1. for i  1 to k do C[i]  0 2.
for j  1 to n
do C[A[ j]]  C[A[ j]] + 1
Q(n) 3.
for i  2 to k
do C[i]  C[i] + C[i–1]
Q(k) 4.
for j  n downto 1
do B[C[A[ j]]]  A[ j]
C[A[ j]]  C[A[ j]] – 1
Q(n)
Q(n + k)

62. What other algorithms have this property?
Stable sorting
Counting sort is a stable sort: it preserves the input order among equal elements. A: 4 1 3 B:
Why this is important?
What other algorithms have this property?

63. Radix sort Similar to sorting the address books
Treat each digit as a key
Start from the least significant bit
Most significant
Least significant

64. Time complexity Sort each of the d digits by counting sort
Total cost: d (n + k)
k = 10
Total cost: Θ(dn)
Partition the d digits into groups of 3
Total cost: (n+103)d/3
We work with binaries rather than decimals
Partition a binary number into groups of r bits
Total cost: (n+2r)d/r
Choose r = log n
Total cost: dn / log n
Compare with dn log n
Catch: faster than quicksort only when n is very large

65. Order statistics Select vs Randomized Select
Note: select is different from selection sort
Algorithm
Worst-case vs average-case (expected) running time analysis
Worst-case Linear-time Select
Algorithm and analysis
Comparison between sorting and select algorithms
Back

66. Randomized select algorithm
RAND-SELECT(A, p, q, i) ⊳ i th smallest of A[ p . . q]
if p = q & i > 1 then error!
r  RAND-PARTITION(A, p, q)
k  r – p + 1 ⊳ k = rank(A[r])
if i = k then return A[ r]
if i < k
then return RAND-SELECT( A, p, r – 1, i )
else return RAND-SELECT( A, r + 1, q, i – k )
£ A[r]
³ A[r] r p q k

67. Complete example: select the 6th smallest element.
i = 6 7 10 5 8 11 3 2 13 3 2 5 7 11 8 10 13
k = 4
i = 6 – 4 = 2 10 8 11 13
k = 3
i = 2 < k
Note: here we always used first element as pivot to do the partition (instead of rand-partition). 8 10
k = 2
i = 2 = k 10

68. Running time of randomized select
T(max(0, n–1)) + n if 0 : n–1 split,
T(max(1, n–2)) + n if 1 : n–2 split, M
T(max(n–1, 0)) + n if n–1 : 0 split,
T(n) ≤
For upper bound, assume ith element always falls in larger side of partition
The expected running time is an average of all cases
Expectation

69. Substitution method Want to show T(n) = O(n).
So need to prove T(n) ≤ cn for n > n0
Assume: T(k) ≤ ck for all k < n
if c ≥ 4
Therefore, T(n) = O(n)

70. Worst-case linear-time select
if i = k then return x
elseif i < k
then recursively SELECT the i th smallest element in the lower part
else recursively SELECT the (i–k)th smallest element in the upper part
SELECT(i, n)
Divide the n elements into groups of 5. Find the median of each 5-element group by rote.
Recursively SELECT the median x of the ën/5û group medians to be the pivot.
Partition around the pivot x. Let k = rank(x).
Same as RAND-SELECT

71. Developing the recurrence
T(n)
if i = k then return x
elseif i < k
then recursively SELECT the i th smallest element in the lower part
else recursively SELECT the (i–k)th smallest element in the upper part
SELECT(i, n)
Divide the n elements into groups of 5. Find the median of each 5-element group by rote.
Recursively SELECT the median x of the ën/5û group medians to be the pivot.
Partition around the pivot x. Let k = rank(x).
Q(n)
T(n/5)
Q(n)
T(7n/10+3)

72. Solving the recurrence
Assumption: T(k) £ ck for all k < n
if n ≥ 60
if c ≥ 20 and n ≥ 60

73. Dynamic programming Proofs of optimal substructure property
Defining recurrence
Solve DP problem without recursive calls
Example DP problems:
Shortest path problem
LCS
Restaurant location
Knapsack
Scheduling
Other similar problems
Back

74. Elements of dynamic programming
Optimal sub-structures
Optimal solutions to the original problem contains optimal solutions to sub-problems
Overlapping sub-problems
Some sub-problems appear in many solutions

75. Two steps to dynamic programming
Formulate the solution as a recurrence relation of solutions to subproblems.
Specify an order to solve the subproblems so you always have what you need.

76. Optimal subpaths
Claim: if a path startgoal is optimal, any sub-path, startx, or xgoal, or xy, where x, y is on the optimal path, is also the shortest.
Proof by contradiction
If the subpath between x and y is not the shortest, we can replace it with the shorter one, which will reduce the total length of the new path => the optimal path from start to goal is not the shortest => contradiction!
Hence, the subpath xy must be the shortest among all paths from x to y
start
goal x y a b c b’
a + b + c is shortest
b’ < b
a + b’ + c < a + b + c

77. Dynamic programming illustration3 9 1 2 3 12 13 15 5 3 3 3 3 3 2 5 2 5 6 8 13 15 2 3 3 9 3 2 4 2 3 7 9 11 13 16 6 2 3 7 4 3 6 3 3 13 11 14 17 20 4 6 3 1 3 1 2 3 2 17 17 17 18 20 G
F(i-1, j) + dist(i-1, j, i, j)
F(i, j) = min
F(i, j-1) + dist(i, j-1, i, j)

78. Trace back 3 9 1 2 3 12 13 15 5 3 3 3 3 3 2 5 2 5 6 8 13 15 2 3 3 9 3 2 4 2 3 7 9 11 13 16 6 2 3 7 4 3 6 3 3 13 11 14 17 20 4 6 3 1 3 1 2 3 2 17 17 17 18 20

79. Longest Common Subsequence
Given two sequences x[1 . . m] and y[1 . . n], find a longest subsequence common to them both.
“a” not “the” x: A B C D y:
BCBA = LCS(x, y)
functional notation, but not a function

80. Optimal substructure
Notice that the LCS problem has optimal substructure: parts of the final solution are solutions of subproblems.
If z = LCS(x, y), then any prefix of z is an LCS of a prefix of x and a prefix of y.
Subproblems: “find LCS of pairs of prefixes of x and y” i m x z n y j

81. Finding length of LCS m x n y
Let c[i, j] be the length of LCS(x[1..i], y[1..j])
=> c[m, n] is the length of LCS(x, y)
If x[m] = y[n]
c[m, n] = c[m-1, n-1] + 1
If x[m] != y[n]
c[m, n] = max { c[m-1, n], c[m, n-1] }

82. DP Algorithm c[i–1, j–1] + 1 if x[i] = y[j],
Key: find out the correct order to solve the sub-problems
Total number of sub-problems: m * n
c[i, j] =
c[i–1, j–1] + 1 if x[i] = y[j],
max{c[i–1, j], c[i, j–1]} otherwise. j n
C(i, j) i m

83. LCS Example (0) ABCB BDCAB X = ABCB; m = |X| = 4j i
Y[j] B D C A B
X[i] A 1 B 2 3 C 4 B
X = ABCB; m = |X| = 4
Y = BDCAB; n = |Y| = 5
Allocate array c[5,6]

84. LCS Example (1) ABCB BDCAB for i = 1 to m c[i,0] = 0j i
Y[j] B D C A B
X[i] A 1 B 2 3 C 4 B
for i = 1 to m c[i,0] = 0
for j = 1 to n c[0,j] = 0

85. LCS Example (2) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 B 2A 1 B 2 3 C 4 B
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

86. LCS Example (3) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 B 2A 1 B 2 3 C 4 B
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

87. LCS Example (4) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1 BA 1 1 B 2 3 C 4 B
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

88. LCS Example (5) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1 1A 1 1 1 B 2 3 C 4 B
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

89. LCS Example (6) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1 1A 1 1 1 B 2 1 3 C 4 B
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

90. LCS Example (7) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1 1A 1 1 1 B 2 1 1 1 1 3 C 4 B
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

91. LCS Example (8) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1 1A 1 1 1 B 2 1 1 1 1 2 3 C 4 B
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

92. LCS Example (9) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1 1A 1 1 1 B 2 1 1 1 1 2 3 C 1 1 4 B
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

93. LCS Example (10) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1A 1 1 1 B 2 1 1 1 1 2 3 C 1 1 2 4 B
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

94. LCS Example (11) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1A 1 1 1 B 2 1 1 1 1 2 3 C 1 1 2 2 2 4 B
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

95. LCS Example (12) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1A 1 1 1 B 2 1 1 1 1 2 3 C 1 1 2 2 2 4 B 1
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

96. LCS Example (13) ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1 1A 1 1 1 B 2 1 1 1 1 2 3 C 1 1 2 2 2 4 B 1 1 2 2
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

97. LCS Example (14) 3 ABCB BDCAB j 0 1 2 3 4 5 i Y[j] B D C A B X[i] A 1A 1 1 1 B 2 1 1 1 1 2 3 C 1 1 2 2 2 3 4 B 1 1 2 2
if ( Xi == Yj )
c[i,j] = c[i-1,j-1] + 1
else c[i,j] = max( c[i-1,j], c[i,j-1] )

98. LCS Algorithm Running Time
LCS algorithm calculates the values of each entry of the array c[m,n]
So what is the running time?
O(m*n)
since each c[i,j] is calculated in constant time, and there are m*n elements in the array

99. How to find actual LCS For example, here
The algorithm just found the length of LCS, but not LCS itself.
How to find the actual LCS?
For each c[i,j] we know how it was acquired:
A match happens only when the first equation is taken
So we can start from c[m,n] and go backwards, remember x[i] whenever c[i,j] = c[i-1, j-1]+1. 2 2
For example, here
c[i,j] = c[i-1,j-1] +1 = 2+1=3 2 3

100. Finding LCS 3 Time for trace back: O(m+n). j 0 1 2 3 4 5 i Y[j] B D C
X[i] A 1 1 1 B 2 1 1 1 1 2 3 C 1 1 2 2 2 3 4 B 1 1 2 2
Time for trace back: O(m+n).

101. Finding LCS (2) 3 LCS (reversed order): B C B B C Bj i
Y[j] B D C A B
X[i] A 1 1 1 B 2 1 1 1 1 2 3 C 1 1 2 2 2 3 4 B 1 1 2 2
LCS (reversed order): B C B
B C B
(this string turned out to be a palindrome)
LCS (straight order):

102. Restaurant location problem
You work in the fast food business
Your company plans to open up new restaurants in Texas along I-35
Towns along the highway called t1, t2, …, tn
Restaurants at ti has estimated annual profit pi
No two restaurants can be located within 10 miles of each other due to some regulation
Your boss wants to maximize the total profit
You want a big bonus
10 mile

103. A DP algorithm Suppose you’ve already found the optimal solution
It will either include tn or not include tn
Case 1: tn not included in optimal solution
Best solution same as best solution for t1 , …, tn-1
Case 2: tn included in optimal solution
Best solution is pn + best solution for t1 , …, tj , where j < n is the largest index so that dist(tj, tn) ≥ 10

104. Recurrence formulation
Let S(i) be the total profit of the optimal solution when the first i towns are considered (not necessarily selected)
S(n) is the optimal solution to the complete problem
S(n-1)
S(j) + pn j < n & dist (tj, tn) ≥ 10
S(n) = max
S(i-1)
S(j) + pi j < i & dist (tj, ti) ≥ 10
S(i) = max
Generalize
Number of sub-problems: n. Boundary condition: S(0) = 0.
Dependency: i
i-1 j S

105. Example S(i-1) S(j) + pi j < i & dist (tj, ti) ≥ 10 S(i) = max
Distance (mi)
100 5 2 2 6 6 3 6 10 7
dummy 7 3 4 12
Profit (100k) 6 7 9 8 3 3 2 4 12 5
S(i) 6 7 9 9 10 12 12 14 26 26
Optimal: 26
S(i-1)
S(j) + pi j < i & dist (tj, ti) ≥ 10
S(i) = max
Natural greedy 1: = 25
Natural greedy 2: = 24

106. Complexity
Time: (nk), where k is the maximum number of towns that are within 10 miles to the left of any town
In the worst case, (n2)
Can be improved to (n) with some preprocessing tricks
Memory: Θ(n)

107. Knapsack problem Each item has a value and a weight
Objective: maximize value
Constraint: knapsack has a weight limitation
Three versions:
0-1 knapsack problem: take each item or leave it
Fractional knapsack problem: items are divisible
Unbounded knapsack problem: unlimited supplies of each item.
Which one is easiest to solve?
We study the 0-1 problem today.

108. Formal definition (0-1 problem)
Knapsack has weight limit W
Items labeled 1, 2, …, n (arbitrarily)
Items have weights w1, w2, …, wn
Assume all weights are integers
For practical reason, only consider wi < W
Items have values v1, v2, …, vn
Objective: find a subset of items, S, such that iS wi  W and iS vi is maximal among all such (feasible) subsets

109. A DP algorithm Suppose you’ve find the optimal solution S
Case 1: item n is included
Case 2: item n is not included
Total weight limit: W
Total weight limit: W wn wn
Find an optimal solution using items 1, 2, …, n-1 with weight limit W - wn
Find an optimal solution using items 1, 2, …, n-1 with weight limit W

110. Recursive formulation
Let V[i, w] be the optimal total value when items 1, 2, …, i are considered for a knapsack with weight limit w
=> V[n, W] is the optimal solution
V[n, W] = max
V[n-1, W-wn] + vn
V[n-1, W]
Generalize
V[i, w] = max
V[i-1, w-wi] + vi item i is taken
V[i-1, w] item i not taken
V[i-1, w] if wi > w item i not taken
Boundary condition: V[i, 0] = 0, V[0, w] = 0. Number of sub-problems = ?

111. Example n = 6 (# of items) W = 10 (weight limit)
Items (weight, value):

112. w 1 2 3 4 5 6 7 8 9 10 i wi vi 1 2 2 2 4 3 wi 3 3 3
V[i-1, w-wi]
V[i-1, w] 4 5 5 6 6
V[i, w] 5 2 4 6 6 9
V[i-1, w-wi] + vi item i is taken
V[i-1, w] item i not taken
max
V[i, w] =
V[i-1, w] if wi > w item i not taken

113. w 1 2 3 4 5 6 7 8 9 10 i wi vi 1 2 4 3 5 6 9 2 2 2 2 2 2 2 2 2 3 5 2 2 3 5 5 5 5 6 8 3 5 2 3 5 6 8 6 8 9 11 2 3 3 6 9 4 6 7 10 12 13 4 7 10 13 15 9 4 4 6 7 10 13
V[i-1, w-wi] + vi item i is taken
V[i-1, w] item i not taken
max
V[i-1, w] if wi > w item i not taken
V[i, w] =

114. w 1 2 3 4 5 6 7 8 9 10 i wi vi 1 2 4 3 5 6 9 2 2 2 2 2 2 2 2 2 3 5 2 2 3 5 5 5 5 6 8 3 5 2 3 5 6 8 6 8 9 11 2 3 3 6 9 4 7 10 12 13 4 6 7 10 13 9 4 4 6 7 10 13 15
Optimal value: 15
Item: 6, 5, 1
Weight: = 10
Value: = 15

115. Time complexity Θ (nW) Polynomial?
Pseudo-polynomial
Works well if W is small
Consider following items (weight, value):
(10, 5), (15, 6), (20, 5), (18, 6)
Weight limit 35
Optimal solution: item 2, 4 (value = 12). Iterate: 2^4 = 16 subsets
Dynamic programming: fill up a 4 x 35 = 140 table entries
What’s the problem?
Many entries are unused: no such weight combination
Top-down may be better

116. Use DP algorithm to solve new problems
Directly map a new problem to a known problem
Modify an algorithm for a similar task
Design your own
Think about the problem recursively
Optimal solution to a larger problem can be computed from the optimal solution of one or more subproblems
These sub-problems can be solved in certain manageable order
Works nicely for naturally ordered data such as strings, trees, some special graphs
Trickier for general graphs
The text book has some very good exercises.

117. Greedy algorithm
Proofs that show locally optimal selection leads to globally optimal solution
Example problems that can be solved by greedy algorithm:
Uniform-profit restaurant location problem
Fractional knap-sack problem
Other similar problems (e.g. event scheduling without weights/values for different event)
Back

118. Greedy algorithm for restaurant location problem
select t1
d = 0;
for (i = 2 to n)
d = d + dist(ti, ti-1);
if (d >= min_dist)
select ti
end 5 2 2 6 6 3 6 10 7 d 5 7 9 15 6 9 15 10 7

119. Complexity Time: Θ(n) Memory: Θ(n) to store the input
Θ(1) for greedy selection

120. Knapsack problem
Each item has a value and a weight
Objective: maximize value
Constraint: knapsack has a weight limitation
Three versions:
0-1 knapsack problem: take each item or leave it
Fractional knapsack problem: items are divisible
Unbounded knapsack problem: unlimited supplies of each item.
Which one is easiest to solve?
We can solve the fractional knapsack problem using greedy algorithm

121. Greedy algorithm for fractional knapsack problem
Compute value/weight ratio for each item
Sort items by their value/weight ratio into decreasing order
Call the remaining item with the highest ratio the most valuable item (MVI)
Iteratively:
If the weight limit can not be reached by adding MVI
Select MVI
Otherwise select MVI partially until weight limit

122. Example Weight limit: 10 9 6 4 2 5 3 1 Value ($) Weight (LB) item 1.5
1.2 1
0.75
$ / LB

123. Example Weight limit: 10 Take item 5 Take item 6 Take 2 LB of item 4
Weight (LB)
Value ($)
$ / LB 5 2 4 6 9
1.5
1.2 1 3
0.75

124. Why is greedy algorithm for fractional knapsack problem valid?
Claim: the optimal solution must contain the MVI as much as possible (either up to the weight limit or until MVI is exhausted)
Proof by contradiction: suppose that the optimal solution does not use all available MVI (i.e., there is still w (w < W) units of MVI left while we choose other items)
We can replace w pounds of less valuable items by MVI
The total weight is the same, but with value higher than the “optimal”
Contradiction w w

125. Good luck with your exam!