1. The father of algorithm analysis
Donald E. Knuth ( )
The father of algorithm analysis
popularizing asymptotic notation.
The author of
TAOCP -- The Art of Computer Programming
Volume 1: Fundamental Algorithms
Volume 2: Seminumerical Algorithms
Volume 3: Sorting and Searching
(among tons of others)
“ I have been a happy man ever since January 1, 1990,
when I no longer had an address.”
11/28/2018
ITK 168

2. The Art of Computer Programming
TAOCP –
The Art of Computer Programming
Volume 1: Fundamental Algorithms
Basic Concepts and Information Structures
Volume 2: Seminumerical Algorithms
Random numbers and Arithmetic
Volume 3: Sorting and Searching
Sorting and Searching
Volume 4: Combinatorial Algorithms
Combinatorial Searching and Recursion
Volume 5: Syntactical Algorithms
Lexical Scanning and Parsing Techniques
Volume 6: Theory of Languages
Mathematical Linguistics
Volume 7: Compiler
Programming Language Translation
Donald E. Knuth
Highlighted in the preface of Vol. 1
11/28/2018
ITK 168

3. Sorting: Arranging items into a certain order.
A collection of items can be stored in a one dimensional array (or linked list, sets or trees, which are more advanced data structure you can learn from ITK179 and ITK279.)
Sometimes, we prefer using one dimensional array due to its random access propertry in the internal memory
11/28/2018
ITK 168

4. Select the right item into the right place.
Selection Sort:
Select the right item into the right place.
The right item
The right place 3 9 5 6 8 1 10 4
swap
11/28/2018
ITK 168

5. Select the right item into the right place.
Selection Sort:
Select the right item into the right place.
The right place
The right item 1 9 5 6 8 3 10 4
swap
11/28/2018
ITK 168

6. Select the right item into the right place.
Selection Sort:
Select the right item into the right place.
The right item
The right place 1 3 5 6 8 9 10 4
swap
11/28/2018
ITK 168

7. Select the right item into the right place.
Selection Sort:
Select the right item into the right place.
The right item
The right place 1 3 4 6 8 9 10 5
swap
11/28/2018
ITK 168

8. Select the right item into the place. 6 12 3 1 10 8 9 7 2 1 12 3 6 10
Selection Sort:
Select the right item into the place. 6 12 3 1 10 8 9 7 2 1 12 3 6 10 8 9 7 2 1 2 3 6 10 8 9 7 12 1 2 3 6 10 8 9 7 12 1 2 3 6 10 8 9 7 12 1 2 3 6 7 8 9 10 12 1 2 3 6 7 8 9 10 12 1 2 3 6 7 8 9 10 12 1 2 3 6 7 8 9 10 12
11/28/2018 1 2 3 6 7 8 9 10 12
ITK 168

9. Selection sort algorithm in Java
public static class SelectionSort {
public void sort(int a[]) {
for (int i = 0; i<a.length-1; i++) { // select one for a[i]
int j = min(a,i);
exchange(a,i,j); }
// select the minimum between a[s] to the end
private int min(int a[], int s) {int m = s;
for (int i=s; i<a.length;i++)
if (a[i] < a[m]) m=i;
return m;
private void exchange(int[] a, int i, int j) {
int temp = a[i];
a[i] = a[j];
a[j] = temp;
} // end of SelectionSort
11/28/2018
ITK 168

10. Insert an item into the right place.
Insertion Sort:
Insert an item into the right place.
sorted 1 3 4 6 8 9 10 12 7
11/28/2018
ITK 168

11. Insert an item into the right place.
Insertion Sort:
Insert an item into the right place.
sorted 1 3 4 6 8 9 10 12 7 2
11/28/2018
ITK 168

12. Insert an item into the right place.
Insertion Sort:
Insert an item into the right place.
sorted 1 3 4 6 8 9 10 12 7 2 7 8 9 10 12 7
11/28/2018
ITK 168

13. Insert an item into the right place. 6 12 3 1 10 8 9 7 2 6 12 3 1 10 8
Insertion Sort:
Insert an item into the right place. 6 12 3 1 10 8 9 7 2 6 12 3 1 10 8 9 7 2 3 6 12 1 10 8 9 7 2 1 3 6 12 10 8 9 7 2 1 3 6 10 12 8 9 7 2 1 3 6 8 10 12 9 7 2 1 3 6 8 9 10 12 7 2 1 3 6 7 8 9 10 12 2 1 2 3 6 7 8 9 10 12
11/28/2018
ITK 168

14. Insertion sort algorithm in Java
public static class InsertionSort {
public void sort(int a[]) {
for (int i = 1;i<a.length;i++)
insert(a,i); // insert a[i] into a[0]..a[i-1] }
private void insert(int a[],int i){//inert a[i] into a[0...i-1]
int v = a[i]; // v is the value to insert
int j;
for (j=i-1; j>=0; j--) {
if (v < a[j]) // a[j] needs to shift down
a[j+1] = a[j];
else break;
a[j+1] = v;
} // end of InsertionSort
11/28/2018
ITK 168

15. Analysis of Insertion/Selection Sorts:
T(n) = n = O(n2)
Shell Sort in
the worst case
O(n3/2)
O(n log n)
Lower bound of sorting algorithms
11/28/2018
ITK 168

16. Sorting, Sorting, Sorting, and more Sorting
ITK 168 students don’t have to “know” the details in the following slides but just “enjoy” the beatuty of algorithms
11/28/2018
ITK 168

17. Quick Sort < < < < < < < < < < < <7 5 8 14 6 1 16 2 11 10 9 13 3 15 12 4
< 7 5 8 4 6 1 3 2 11 10 9 13 16 15 12 14
<
<
< 2 3 1 4 6 8 5 7 11 10 9 12 16 15 13 14
<
<
<
<
<
<
< 2 1 3 4 6 5 8 7 9 10 11 12 14 13 15 16
<
<
<
<
<
<
<
<
<
<
<
<
<
<
< 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
11/28/2018
ITK 168

18. Quick Sort 7 5 8 14 6 1 16 2 11 10 9 13 3 15 12 4 4 3 2 16 14 8 2 5 4 3 6 1 7 16 11 10 9 13 14 15 12 8 1 2 4 3 6 5 7 8 11 10 9 13 14 15 12 16 1 2 3 4 6 5 7 8 11 10 9 13 14 15 12 16 1 2 3 4 5 6 7 8 9 10 11 13 14 15 12 16 1 2 3 4 5 6 7 8 9 10 11 12 13 15 14 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
11/28/2018
ITK 168

19. Quick Sort /* Quick Sort */
private void SwapTwo(int A[], int i, int j){
int temp = A[i];
A[i]=A[j];
A[j]=temp; }
public void QuickSort(int A[], int h, int t){ // h: head
// t: tail
if (h == t) return;
int i=h+1, j=t;
if (i==j) {
if (A[h] > A[i]) SwapTwo(A,h,i);
return;
while (i < j) {
while (A[h] >= A[i] && i < t) i++;
while (A[h] <= A[j] && j > h) j--;
if (i < j) SwapTwo(A,i++,j--);
if (i==j && A[h]>A[i]) SwapTwo(A,h,i);
QuickSort(A,h,i-1);
QuickSort(A,i,t);
Quick Sort
11/28/2018
ITK 168

20. Mergesort
Merge two sorted lists 3 5 7 8 1 2 6
O(n)
11/28/2018
ITK 168

21. 7 5 3 8 2 1 6
Merge Sort 7 5 3 8 2 1 6 7 5 3 8 2 1 6
O(log n) 7 5 3 8 2 1 5 7 3 8 1 2
O(n log n) 3 5 7 8 1 2 6 1 2 3 5 6 7 8
11/28/2018
ITK 168

22. Incremental insertion (selection) sort. Shell Sort: h = 47 5 8 14 6 1 16 2 11 10 9 13 3 15 12 4 3 5 8 14 6 1 16 2 7 10 9 13 11 15 12 4 3 1 8 14 6 5 16 2 7 10 9 13 11 15 12 4 3 1 8 14 6 5 9 2 7 10 12 13 11 15 16 4 3 1 8 2 6 5 9 4 7 10 12 13 11 15 16 14
11/28/2018
ITK 168

23. Shell Sort: hk-sorted h = 4 4-sorted 2-sorted 1-sorted 3 1 8 2 6 5 9 47 10 12 13 11 15 16 14
4-sorted 3 1 6 2 7 4 8 5 9 10 11 13 12 14 16 15
2-sorted 1 2 3 4 5 6 7 8 9 10 11 12 13 14 15 16
1-sorted
11/28/2018
ITK 168

24. Incremental insertion
Shell Sort:
Incremental insertion
with
Increment sequence,
hk-sorted
1-sorted
ht-sorted
ht-1-sorted
hk-1-sorted  (implies) hk-sorted
hk-1-sorted
hk-sorted
Shell suggested:
Not a good suggestion, worst case: O(n2)
11/28/2018
ITK 168

25. Shell sort algorithm in Java
public static class ShellSort {
public void sort(int a[]) {
int h = a.length/2;
while (h > 0) { // h-sort
for (int i=h; i<a.length;i++) {
int t=a[i], j=i;
while (j >= h && a[j-h] > t) {
a[j] = a[j-h];
j -= h; }
a[j]=t;
h /= 2;
} // end increment
} // end of ShellSort
Shell sort algorithm in Java
bad strategy
h = 4
h = 4
h = 4 i
h = 4 j j j j 7 5 8 14 6 12 9 2 16 1
11/28/2018
ITK 168

26. Shell Sort: using Shell’s suggestion
Worst case analysis: O(n2)
Idea:
Right before the final sort, let the smallest n/2 be distributed in the even position.
e.g.
2-sorted 9 1 10 2 11 3 12 4 13 5 14 6 15 7 16 8 1 2 3 4 5 6 7 8
final sort
11/28/2018
ITK 168

27. Shell Sort: using Shell’s suggestion
What kind of initial array will result in such 2-sorted? 9 1 10 2 11 3 12 4 13 5 14 6 15 7 16 8
Let Ah[s] = A[s], A[s+h], A[s+2h], ....
Ah[1]
Ah[0]
even positions
odd positions 5 13 1 11 3
Ah[2] 12 7 4 15 9 10 8
Ah[3] 6 2 14 16
Ah[5]
Ah[6]
Ah[4]
Ah[7]
elements will never cross the even-/odd-cell before the final sort
8-sort ,
4-sort ,
2-sort
11/28/2018
ITK 168

28. Shell Sort:
hk-sorted 7 5 8 14 6 1 16 2 11 10 9 13 3 15 12 4
3-sorted 3 2 1 4 5 8 7 6 11 10 9 12 14 15 13 16
A[p-6]
A[p] 6
2-sorted 1 2 3 4 5 6 7 8 9 10 11 12 13 15 14 16 6 6
A[p-6]
A[p]
A[p-6]
A[p]
A[p-(x2+y3)] < A[p]
11/28/2018
ITK 168

29. Percolating a non-heap Worst case: O(n)
Heapsort
Percolating a non-heap
Worst case: O(n) 34 2 15 2 5 7 30 7 15 2 6 23 5 9 21 25 30 7 6 23 9 21 6 15 32 5 23 40 11 31 30 21 7 25 15 32 34 40 11 31 30 25
11/28/2018
ITK 168

30. Percolating a non-heap
Heapsort
O(log n) 2
Percolating a non-heap
O(log n-1) 5
Worst case: O(n)
O(log n-2) 6 7 2 5 7 6 23 9 21 40
O(log 1) 15 32 34 40 11 31 30 25
O(n+n log n) = O(n log n)
11/28/2018
ITK 168