1. Analysis of Algorithms CS 477/677 Instructor: Monica Nicolescu Lecture 5

2. CS 477/677 - Lecture 52 The Sorting Problem Input: –A sequence of n numbers a 1, a 2,..., a n Output: –A permutation (reordering) a 1 ’, a 2 ’,..., a n ’ of the input sequence such that a 1 ’ ≤ a 2 ’ ≤ · · · ≤ a n ’

3. CS 477/677 - Lecture 53 Why Study Sorting Algorithms? There are a variety of situations that we can encounter –Do we have randomly ordered keys? –Are all keys distinct? –How large is the set of keys to be ordered? –Need guaranteed performance? –Does the algorithm sort in place? –Is the algorithm stable? Various algorithms are better suited to some of these situations

4. CS 477/677 - Lecture 54 Stability A STABLE sort preserves relative order of records with equal keys Sort file on first key: Sort file on second key: Records with key value 3 are not in order on first key!!

5. CS 477/677 - Lecture 55 Insertion Sort Idea: like sorting a hand of playing cards –Start with an empty left hand and the cards facing down on the table –Remove one card at a time from the table, and insert it into the correct position in the left hand compare it with each of the cards already in the hand, from right to left –The cards held in the left hand are sorted these cards were originally the top cards of the pile on the table

6. CS 477/677 - Lecture 56 Example

7. CS 477/677 - Lecture 57 INSERTION-SORT Alg.: INSERTION-SORT(A) for j ← 2 to n do key ← A[ j ] Insert A[ j ] into the sorted sequence A[1.. j -1] i ← j - 1 while i > 0 and A[i] > key do A[i + 1] ← A[i] i ← i – 1 A[i + 1] ← key Insertion sort – sorts the elements in place a8a8 a7a7 a6a6 a5a5 a4a4 a3a3 a2a2 a1a1 12345678 key

8. CS 477/677 - Lecture 58 Loop Invariant for Insertion Sort Alg.: INSERTION-SORT(A) for j ← 2 to n do key ← A[ j ] Insert A[ j ] into the sorted sequence A[1.. j -1] i ← j - 1 while i > 0 and A[i] > key do A[i + 1] ← A[i] i ← i – 1 A[i + 1] ← key Invariant: at the start of each iteration of the for loop, the elements in A[1.. j-1] are in sorted order

9. CS 477/677 - Lecture 59 Proving Loop Invariants Proving loop invariants works like induction Initialization (base case): –It is true prior to the first iteration of the loop Maintenance (inductive step): –If it is true before an iteration of the loop, it remains true before the next iteration Termination: –When the loop terminates, the invariant gives us a useful property that helps show that the algorithm is correct

10. CS 477/677 - Lecture 510 Loop Invariant for Insertion Sort Initialization: –Just before the first iteration, j = 2 : the subarray A[1.. j-1] = A[1], (the element originally in A[1] ) – is sorted

11. CS 477/677 - Lecture 511 Loop Invariant for Insertion Sort Maintenance: –the while inner loop moves A[j -1], A[j -2], A[j -3], and so on, by one position to the right until the proper position for key (which has the value that started out in A[j] ) is found –At that point, the value of key is placed into this position.

12. CS 477/677 - Lecture 512 Loop Invariant for Insertion Sort Termination: –The outer for loop ends when j = n + 1  j-1 = n –Replace n with j-1 in the loop invariant: the subarray A[1.. n] consists of the elements originally in A[1.. n], but in sorted order The entire array is sorted! jj - 1

13. CS 477/677 - Lecture 513 Analysis of Insertion Sort cost times c 1 n c 2 n-1 0 n-1 c 4 n-1 c 5 c 6 c 7 c 8 n-1 INSERTION-SORT(A) for j ← 2 to n do key ← A[ j ] Insert A[ j ] into the sorted sequence A[1.. j -1] i ← j - 1 while i > 0 and A[i] > key do A[i + 1] ← A[i] i ← i – 1 A[i + 1] ← key

14. CS 477/677 - Lecture 514 Best Case Analysis The array is already sorted –A[i] ≤ key upon the first time the while loop test is run (when i = j -1) –t j = 1 T(n) = c 1 n + c 2 (n -1) + c 4 (n -1) + c 5 (n -1) + c 8 (n-1) = (c 1 + c 2 + c 4 + c 5 + c 8 )n - (c 2 + c 4 + c 5 + c 8 ) = an + b =  (n) “while i > 0 and A[i] > key”

15. CS 477/677 - Lecture 515 Worst Case Analysis The array is in reverse sorted order –Always A[i] > key in while loop test –Have to compare key with all elements to the left of the j -th position  compare with j-1 elements  t j = j a quadratic function of n T(n) =  (n 2 ) order of growth in n 2 “while i > 0 and A[i] > key”

16. CS 477/677 - Lecture 516 Comparisons and Exchanges in Insertion Sort INSERTION-SORT(A) for j ← 2 to n do key ← A[ j ] Insert A[ j ] into the sorted sequence A[1.. j -1] i ← j - 1 while i > 0 and A[i] > key do A[i + 1] ← A[i] i ← i – 1 A[i + 1] ← key cost times c 1 n c 2 n-1 0 n-1 c 4 n-1 c 5 c 6 c 7 c 8 n-1  n 2 /2 comparisons  n 2 /2 exchanges

17. CS 477/677 - Lecture 517 Insertion Sort - Summary Idea: like sorting a hand of playing cards –Start with an empty left hand and the cards facing down on the table. –Remove one card at a time from the table, and insert it into the correct position in the left hand Advantages –Good running time for “almost sorted” arrays  (n) Disadvantages –  (n 2 ) running time in worst and average case –  n 2 /2 comparisons and n 2 /2 exchanges

18. CS 477/677 - Lecture 518 Bubble Sort Idea: –Repeatedly pass through the array –Swaps adjacent elements that are out of order Easier to implement, but slower than Insertion sort 123n i 1329648 j

19. CS 477/677 - Lecture 519 Example 1329648 i = 1j 3129648 j 3219648 j 3291648 j 3296148 j 3296418 j 3296481 j 3296481 i = 2j 3964821 i = 3j 9648321 i = 4j 9684321 i = 5j 9864321 i = 6j 9864321 i = 7 j

20. CS 477/677 - Lecture 520 Bubble Sort Alg.: BUBBLESORT(A) for i  1 to length[A] do for j  length[A] downto i + 1 do if A[j] < A[j -1] then exchange A[j]  A[j-1] 1329648 i = 1j i

21. CS 477/677 - Lecture 521 Bubble-Sort Running Time T(n) =  (n 2 ) Alg.: BUBBLESORT(A) for i  1 to length[A] do for j  length[A] downto i + 1 do if A[j] < A[j -1] then exchange A[j]  A[j-1] T(n) =c 1 (n+1) +c2c2 c3c3 c4c4 =  (n) + (c 2 + c 3 + c 4 ) Comparisons:  n 2 /2 Exchanges:  n 2 /2

22. CS 477/677 - Lecture 522 Selection Sort Idea: –Find the smallest element in the array –Exchange it with the element in the first position –Find the second smallest element and exchange it with the element in the second position –Continue until the array is sorted Invariant: –All elements to the left of the current index are in sorted order and never changed again Disadvantage: –Running time depends only slightly on the amount of order in the file 1329648

23. CS 477/677 - Lecture 523 Example 1329648 8329641 8349621 8649321 8964321 8694321 9864321 9864321

24. CS 477/677 - Lecture 524 Selection Sort Alg.: SELECTION-SORT(A) n ← length[A] for j ← 1 to n - 1 do smallest ← j for i ← j + 1 to n do if A[i] < A[smallest] then smallest ← i exchange A[j] ↔ A[smallest] 1329648

25. CS 477/677 - Lecture 525  n 2 /2 comparisons Analysis of Selection Sort Alg.: SELECTION-SORT(A) n ← length[A] for j ← 1 to n - 1 do smallest ← j for i ← j + 1 to n do if A[i] < A[smallest] then smallest ← i exchange A[j] ↔ A[smallest] cost times c 1 1 c 2 n c 3 n-1 c 4 c 5 c 6 c 7 n-1  n exchanges T(n) =  (n 2 )

26. CS 477/677 - Lecture 526 Divide-and-Conquer Divide the problem into a number of subproblems –Similar sub-problems of smaller size Conquer the sub-problems –Solve the sub-problems recursively –Sub-problem size small enough  solve the problems in straightforward manner Combine the solutions to the sub-problems –Obtain the solution for the original problem

27. CS 477/677 - Lecture 527 Merge Sort Approach To sort an array A[p.. r]: Divide –Divide the n-element sequence to be sorted into two subsequences of n/2 elements each Conquer –Sort the subsequences recursively using merge sort –When the size of the sequences is 1 there is nothing more to do Combine –Merge the two sorted subsequences

28. CS 477/677 - Lecture 528 Merge Sort Alg.: MERGE-SORT (A, p, r) if p < r Check for base case then q ←  (p + r)/2  Divide MERGE-SORT (A, p, q) Conquer MERGE-SORT (A, q + 1, r) Conquer MERGE (A, p, q, r) Combine Initial call: MERGE-SORT (A, 1, n) 12345678 6231742 5 p r q

29. CS 477/677 - Lecture 529 Example – n Power of 2 12345678 q = 4 6231742 5 1234 742 5 5678 623 1 12 2 5 34 74 56 3 1 78 62 1 5 2 2 3 4 4 7 1 6 3 7 2 8 6 5 Example

30. CS 477/677 - Lecture 530 Merging Input: Array A and indices p, q, r such that p ≤ q < r –Subarrays A[p.. q] and A[q + 1.. r] are sorted Output: One single sorted subarray A[p.. r] 12345678 6321754 2 p r q

31. CS 477/677 - Lecture 531 Merging Idea for merging: –Two piles of sorted cards Choose the smaller of the two top cards Remove it and place it in the output pile –Repeat the process until one pile is empty –Take the remaining input pile and place it face-down onto the output pile

32. CS 477/677 - Lecture 532 Merge - Pseudocode Alg.: MERGE(A, p, q, r) 1.Compute n 1 and n 2 2.Copy the first n 1 elements into L[1.. n 1 + 1] and the next n 2 elements into R[1.. n 2 + 1] 3.L[n 1 + 1] ←  ; R[n 2 + 1] ←  4. i ← 1; j ← 1 5. for k ← p to r 6. do if L[ i ] ≤ R[ j ] 7. then A[k] ← L[ i ] 8. i ← i + 1 9. else A[k] ← R[ j ] 10. j ← j + 1 pq 7542 6321 rq + 1 L R   12345678 6321754 2 p r q n1n1 n2n2

33. CS 477/677 - Lecture 533 Running Time of Merge Initialization (copying into temporary arrays): –  (n 1 + n 2 ) =  (n) Adding the elements to the final array (the for loop): –n iterations, each taking constant time   (n) Total time for Merge: –  (n)

34. CS 477/677 - Lecture 534 Analyzing Divide and Conquer Algorithms The recurrence is based on the three steps of the paradigm: –T(n) – running time on a problem of size n –Divide the problem into a subproblems, each of size n/b: takes D(n) –Conquer (solve) the subproblems: takes aT(n/b) –Combine the solutions: takes C(n)  (1) if n ≤ c T(n) = aT(n/b) + D(n) + C(n) otherwise

35. CS 477/677 - Lecture 535 MERGE – SORT Running Time Divide: –compute q as the average of p and r: D(n) =  (1) Conquer: –recursively solve 2 subproblems, each of size n/2  2T (n/2) Combine: –MERGE on an n -element subarray takes  (n) time  C(n) =  (n)  (1) if n =1 T(n) = 2T(n/2) +  (n) if n > 1

36. CS 477/677 - Lecture 536 Solve the Recurrence T(n) = cif n = 1 2T(n/2) + cnif n > 1 Use Master’s Theorem: Compare n with f(n) = cn Case 2: T(n) = Θ(nlgn)

37. CS 477/677 - Lecture 537 Merge Sort - Discussion Running time insensitive of the input Advantages: –Guaranteed to run in  (nlgn) Disadvantage –Requires extra space  N Applications –Maintain a large ordered data file –How would you use Merge sort to do this?

38. CS 477/677 - Lecture 538 Readings Chapter 2