1. Sorting & Lower Bounds Jeff Edmonds York University COSC 3101 Lecture 5

2. Problem 1: Replace... while x!=0 AND y!=0 AND z!=0 do with... while x!=0 OR y!=0 OR z!=0 do Now available on website. DUE: February 13th CORRECTION: Assignment 2

3. Review of Sorting Algorithms

4. Minimal effort splitting Lots of effort recombining Lots of effort splitting Minimal effort recombining Merge SortInsertion Sort Quick SortSelection Sort Size of Sublists n /2,n /2 n-1,1 Four Recursive Sorts

5. Selection Sort 5, 9, 14, 31, 25, 8, 18 Swap with first item 25, 9, 14, 31, 5, 8, 18 Search array for minimum item 5, 9, 14, 31, 25, 8, 18 Search remaining array for min item 5, 8, 14, 31, 25, 9, 18 Swap with first item … Loop … 5, 8, 9, 14, 18, 25, 31

6. Insertion Sort 25, 9, 14, 31, 5, 8, 18 Insert 1 st item in correct position … Loop … 9, 25, 14, 31, 5, 8, 18 Insert 2 nd item in correct position 9, 14, 25, 31, 5, 8, 18 Insert 3 rd item in correct position 5, 8, 9, 14, 18, 25, 31

7. Selection Sort Average & Worst Time:  (n 2 ) Insertion Sort Average & Worst Time:  (n 2 )

8. Merge Sort 88 14 98 25 62 52 79 30 23 31 Divide and Conquer

9. Merge Sort 88 14 98 25 62 52 79 30 23 31 Split Set into Two (no real work) 25,31,52,88,98 Get one friend to sort the first half. 14,23,30,62,79 Get one friend to sort the second half.

10. Merge Sort Merge two sorted lists into one 25,31,52,88,9814,23,30,62,7914,23,25,30,31,52,62,79,88,98

12. Merge Sort Time: T(n) = =  (n log n) 2T(n/2) +  (n)

13. Quick Sort 88 14 98 25 62 52 79 30 23 31 Divide and Conquer

14. Quick Sort 88 14 98 25 62 52 79 30 23 31 Partition set into two using randomly chosen pivot 14 25 30 23 31 88 98 62 79 ≤ 52 ≤

15. Quick Sort 14 25 30 23 31 88 98 62 79 ≤ 52 ≤ 14,23,25,30,31 Get one friend to sort the first half. 62,79,98,88 Get one friend to sort the second half.

16. Quick Sort 14,23,25,30,31 62,79,98,8852 Glue pieces together. (No real work) 14,23,25,30,31,52,62,79,88,98

18. Quick Sort 88 14 98 25 62 52 79 30 23 31 Let pivot be the first element in the list? 14 25 30 23 88 98 62 79 ≤ 31 ≤ 52

19. Quick Sort ≤ 14 ≤ 14,23,25,30,31,52,62,79,88,98 23,25,30,31,52,62,79,88,98 If the list is already sorted, then the list is worst case unbalanced.

20. Quick Sort Best Time: Worst Time: Expected Time: T(n) = 2T(n/2) +  (n) =  (n log(n))

21. Quick Sort T(n) = 2T(n/2) +  (n) =  (n log(n)) Best Time: Worst Time: Expected Time: =  (n 2 ) T(n) = T(0) + T(n-1) +  (n)

22. Quick Sort T(n) = 2T(n/2) +  (n) =  (n log(n)) Best Time: T(n) = T(0) + T(n-1) +  (n) Worst Time: Expected Time: =  (n 2 ) T(n) = T( 1 / 3 n) + T( 2 / 3 n) +  (n) =  (n log(n))

23. Heaps, Heap Sort, & Priority Queues

24. Heap Definition Completely Balanced Binary Tree The value of each node  each of the node's children. Left or right child could be larger. Where can 1 go? Maximum is at root. Where can 8 go? Where can 9 go?

25. Heap Data Structure Completely Balanced Binary Tree Implemented by an Array

26. Make Heap Get help from friends

27. Heapify ? Maximum is at root. Where is the maximum?

28. Find the maximum. Put it in place ? Repeat Heapify

29. Heap Running Time: Heapify

32. Make Heap Get help from friends T(n) = 2T(n/2) + log(n) Running time: =  (n)

33. Heaps Heap ?

34. ?

35. ?

36. ?

38. Running Time: i log(n) -i 2 log(n) -i

39. Insertion Sort Largest i values are sorted on side. Remaining values are off to side. 6,7,8,9 < 3 4 1 5 2 Max is easier to find if a heap. Insertion

40. Heap Sort Largest i values are sorted on side. Remaining values are in a heap.

41. Heap Data Structure HeapArray 5 34 21 9 8 7 6 Heap Array

42. Heap Sort Largest i values are sorted on side. Remaining values are in a heap Put next value where it belongs. Heap ?

43. Heap Sort ? ?? ?? ?? Heap

44. Heap Sort

45. Running Time:

46. Priority Queues Maintains dynamic set, S, of elements, each with a key. Max-priority queue supports: INSERT(S,x) MAXIMUM(S) EXTRACT-MAX(S) INCREASE-KEY(S,x,k) Application: Shedule jobs on a shared computer.

47. Priority Queues cont’d... MAXIMUM(S): EXTRACT-MAX(S):

48. Priority Queues cont’d... INSERT(S,x): INCREASE-KEY(S,x,k):

49. Other Sorting Algorithms: Counting Sort Radix Sort Bucket Sort

50. Counting Sort Input: Array A[1,…,n], with all elements in {1,..,k}. Output: Sorted array B[1,..,n]. Auxiliary Storage: C[1,..,k]. 5 1 5 3 3 2 A = _ _ _ B = _ _ _ _ _ C =

51. Counting Sort Example 5 1 5 3 3 2 A = 0 0 0 0 0 C = 1 1 2 0 2 C = 1 2 4 4 6 C =

52. Counting Sort Example cont’d 5 1 5 3 3 2 A = _ 2 _ _ _ _ B = 1 2 4 4 6 C = 1 _ 2 _ 3 _ _ B = 1 1 4 4 6 C = 3 _ 2 3 3 _ _ B = 1 1 3 4 6 C = 2

53. Counting Sort Example cont’d 5 1 5 3 3 2 A = _ 2 3 3 _ 5 B = 1 1 2 4 6 C = 5 1 2 3 3 _ 5 B = 1 1 2 4 5 C = 0 1 2 3 3 5 5 B = 0 1 2 4 5 C = 4

54. Counting Sort Analysis  (k)

55. Counting Sort Analysis  (k)  (n)

56. Counting Sort Analysis  (k)  (n)

57. Counting Sort Analysis  (k)  (n)

58. Counting Sort Analysis  (k)  (n)  (n + k) =  (n) if k = O(n)

59. Radix Sort Sort numbers by columns, starting with least significant digit (radix):

60. Radix Sort Analysis Assume use of Counting Sort (  (n)) in each pass. For constant d digits, overall  (dn). Counting sort not in place, therefore may prefer a comparison sort algorithm.

61. Bucket Sort Assumes input elements distributed uniformly on [0,1). List elements from buckets in order. Sort each bucket. Distribute n input values into the buckets. Divide [0,1) into equal sized buckets.

62. Bucket Sort Example

63. Bucket Sort Analysis Division into buckets & concatentation:  (n) Insertion sort:  (n 2 ) Define n i = number of elements in bucket B[i]. T(n) =  (n) +  i=1 O(n i 2 ) Take expectation of both sides: E[T(n)] =  (n) +  i=1 E[O(n i 2 )] =  (n) + n O(2 - 1/n) =  (n) n-1 2 - 1/n

64. Lower Bounds for Sorting using Information Theory

65. The Time Complexity of a Problem P Merge, Quick, and Heap Sort can sort N numbers using O(N log N) comparisons between the values. Theorem: No algorithm can sort faster. The Time Complexity of a Problem P: The minimum time needed by an algorithm to solve it.

66. The Time Complexity of a Problem P The minimum time needed by an algorithm to solve it. Problem P is computable in time T upper (n) if there is an algorithm A which outputs the correct answer in this much time Eg: Sorting computable in T upper (n) = O(n 2 ) time.  A,  I, A(I)=P(I) and Time(A,I)  T upper (|I|) Upper Bound:

67. Understand Quantifiers!!! One girl Could be a separate girl for each boy. SamMary BobBeth John Marilin Monro FredAnn SamMary BobBeth John Marilin Monro FredAnn  A,  I, A(I)=P(I) and Time(A,I)  T upper (|I|)

68. The Time Complexity of a Problem P The minimum time needed by an algorithm to solve it.  A,  I, A(I)=P(I) and Time(A,I)  T upper (|I|)  I,  A, A(I)=P(I) and Time(A,I)  T upper (|I|) What does this say? True for any problem P and time T upper. Given fixed I, its output is P(I).

69. The Time Complexity of a Problem P The minimum time needed by an algorithm to solve it. Time T lower (n) is a lower bound for problem p if no algorithm solves the problem faster. There may be algorithms that give the correct answer or run quickly on some inputs instance. Lower Bound:

70. The Time Complexity of a Problem P The minimum time needed by an algorithm to solve it. Lower Bound: Time T lower (n) is a lower bound for problem p if no algorithm solves the problem faster. eg: No algorithm can sort N values in T lower = sqrt(N) time. But for every algorithm, there is at least one instance I for which either the algorithm gives the wrong answer or it runs in too much time.  A,  I, A(I)  P(I) or Time(A,I)  T lower (|I|)

71.  A,  I, A(I)=P(I) and Time(A,I)  T upper (|I|) Lower Bound: Upper Bound: The Time Complexity of a Problem P The minimum time needed by an algorithm to solve it. “There is” and “there isn’t a faster algorithm” are almost negations of each other.

72.  A,  I, A(I)=P(I) and Time(A,I)  T upper (|I|) Upper Bound: Prover-Adversary Game I have an algorithm A that I claim works and is fast. I win if A on input I gives the correct output in the allotted time. Oh yeah, I have an input I for which it does not. What we have been doing all along.

73. Lower Bound: Prover-Adversary Game I win if A on input I gives the wrong output or runs slow.  A,  I, A(I)  P(I) or Time(A,I)  T lower (|I|) Proof by contradiction. I have an algorithm A that I claim works and is fast. Oh yeah, I have an input I for which it does not.

74. Lower Bound: Prover-Adversary Game  A,  I, A(I)  P(I) or Time(A,I)  T lower (|I|) I have an algorithm A that I claim works and is fast. Lower bounds are very hard to prove, because I must consider every algorithm no matter how strange.

75. The Yes/No Questions Game I choose a number  [1..N]. I ask you yes/no questions. Time is the number of questions asked in the worst case. I answer. I determine the number.

76. The Yes/No Questions Game Upper Bound 6 Great!

77. The Yes/No Questions Game Upper Bound Time N leaves = # of questions = height = log 2 (N)

78. The Yes/No Questions Game Lower Bound? Time N leaves Is there a faster algorithm? Is the third bit 1? If different questions? = # of questions = height = log 2 (N)

79. The Yes/No Questions Game Lower Bound? N leaves Is there a faster algorithm? If it has a different structure? Best case Worst case Time = # of questions = height = log 2 (N)

80. The Yes/No Questions Game Lower Bound Theorem: For every question strategy A, with the worst case object I to ask about, log 2 N questions need to be asked to determine one of N objects.  A,  I, A(I)  P(I) or Time(A,I)  log 2 N Proof: Prover/Adversary Game

81. Lower Bound Proof Oh yeah, I have an input I for which it does not. I have an algorithm A that I claim works and is fast. Two cases: # output leaves < N # output leaves  N

82. I win if A on input I gives the wrong output or requires log N questions. I have an algorithm A that I claim works and is fast. Man your algorithm does not have N output leaves. Lower Bound Proof: case 1 I give a input I = 4 with missing output.

83. I win if A on input I gives the wrong output or requires log N questions. Good now your algorithm has all N outputs as leaves. It must have height  log N. I have an algorithm A that I claim works and is fast. I give a input I = 5 at a deep leaf. Lower Bound Proof: case 2

84. The Yes/No Questions Game Lower Bound Theorem: For every question strategy A, with the worst case object I to ask about, log 2 N questions need to be asked to determine one of N objects.  A,  I, A(I)  P(I) or Time(A,I)  log 2 N Proof: Prover/Adversary Game End of Proof.

85. Communication Complexity Or a obj  a set of N objs. Time is the number of bits sent in the worst case. I send you a stream of bits. I determine the object. I choose a number  [1..N].

86. Communication Complexity Upper Bound 6 Great!

87. Communication Complexity Lower Bound Theorem: For every communication strategy A, with the worst case object I to communicate, log 2 N bits need to transmitted to communicate one of N objects.  A,  I, A(I)  P(I) or Time(A,I)  log 2 N

88. The Sorting Game I choose a permutation of {1,2,3,…,N}. I ask you yes/no questions. Time is the number of questions asked in the worst case. I answer. I determine the permuation.

89. Sorting Upper Bound b,c,a Great!

90. Time N! leaves = # of questions = height = log 2 (N!) Sorting Upper Bound

91. N! = 1 × 2 × 3 × … × N / 2 × … × N N factors each at most N. N / 2 factors each at least N / 2.  N N N / 2  N/2N/2 Bounding log(N!) N / 2 log( N / 2 )  log(N!)  N log(N) =  (N log(N)).

92. Time N! leaves = # of questions = height = log 2 (N!) =  (N log(N)). Is there a faster algorithm? Is the third bit of c 1? If different questions? Sorting Lower Bound

93. Is there a faster algorithm? What if a different model of computation? class InsertionSortAlgorithm { for (int i = 1; i < A.length; i++) { int j = i; int B = A[i]; while ((j > 0) && (A[j-1] > A[i])) { A[j] = A[j-1]; j--; } A[j] = B; }} Input: A = [a, b, c]Output in increasing order. Sorting - Lower Bound

94. I choose a permutation of {1,2,3,…,N}. I answer. I determine the permuation. class InsertionSortAlgorithm { for (int i = 1; i < A.length; i++) { int j = i; int B = A[i]; while ((j > 0) && (A[j-1] > A[i])) { A[j] = A[j-1]; j--; } A[j] = B; }} I ask you this yes/no question. I run my algorithm until I need to know something about the input permutation. Sorting with Alice & Bob

95. Theorem: For every sorting algorithm A, with the worst case input instance I,  (N log 2 N) comparisons (or other bit operations) need to be executed to sort N objects.  A,  I, A(I)  P(I) or Time(A,I)  N log 2 N Proof: Prover/Adversary Game Sorting - Lower Bound

96. Lower Bound Proof Oh yeah, I have an input I for which it does not. I have an algorithm A that I claim works and is fast. Two cases: # output leaves < N! # output leaves  N!

97. I win if A on input I gives the wrong output or requires log N questions. Man your algorithm does not have N! output leaves. Lower Bound Proof: case 1 I have an algorithm A that I claim works and is fast. I give a input I = with missing output. b,a,c

98. I win if A on input I gives the wrong output or requires log N! questions. Good now your algorithm has all N! outputs as leaves. It must have height  log N!. I have an algorithm A that I claim works and is fast. Lower Bound Proof: case 2 =  (N log(N)). I give a input I = at a deep leaf.

99. Sorting Lower Bound Theorem: For every sorting algorithm A, on the worst case input instance I,  (N log 2 N) comparisons (or other bit operations) need to be executed to sort N objects.  A,  I, A(I)  P(I) or Time(A,I)  N log 2 N Proof: Prover/Adversary Game End of Proof.

100. End

101. Problem 1: Replace... while x!=0 AND y!=0 AND z!=0 do with... while x!=0 OR y!=0 OR z!=0 do Now available on website. DUE: February 13th CORRECTION: Don’t forget.... Assignment 2