1. The Time Complexity of an Algorithm Specifies how the running time depends on the size of the input. CSE 3101Z Design and Analysis of Algorithms

2. COSC 3101, PROF. J. ELDER 2 Purpose To estimate how long a program will run. To estimate the largest input that can reasonably be given to the program. To compare the efficiency of different algorithms. To help focus on the parts of code that are executed the largest number of times. To choose an algorithm for an application.

3. COSC 3101, PROF. J. ELDER 3 Time Complexity Is a Function Specifies how the running time depends on the size of the input. A function mapping “size” of input  “time” T(n) executed.

4. COSC 3101, PROF. J. ELDER 4 Example: Insertion Sort

5. COSC 3101, PROF. J. ELDER 5 Example: Insertion Sort

6. COSC 3101, PROF. J. ELDER 6 Example: Insertion Sort

7. COSC 3101, PROF. J. ELDER 7 Example: Merge Sort

8. COSC 3101, PROF. J. ELDER 8

9. 9 Example: Merge Sort

10. COSC 3101, PROF. J. ELDER 10 Example: Merge Sort

11. Relevant Mathematics: Classifying Functions Input Size Time

12. COSC 3101, PROF. J. ELDER 12 Assumed Knowledge See J. Edmonds, How to Think About Algorithms (HTA): http://www.cse.yorku.ca/~jeff/courses/3101/ Chapter 22. Existential and Universal Quantifiers Chapter 24. Logarithms and Exponentials

13. Classifying Functions Describing how fast a function grows without going into too much detail.

14. COSC 3101, PROF. J. ELDER 14 Which are more alike?

15. COSC 3101, PROF. J. ELDER 15 Which are more alike? Mammals

16. COSC 3101, PROF. J. ELDER 16 Which are more alike?

17. COSC 3101, PROF. J. ELDER 17 Which are more alike? Dogs

18. COSC 3101, PROF. J. ELDER 18 Classifying Animals Vertebrates BirdsMammals Reptiles Fish Dogs Giraffe

19. COSC 3101, PROF. J. ELDER 19 Classifying Functions Note: The universe is estimated to contain ~10 80 particles. T(n)T(n)101001,00010,000 log n 36913 n 1/2 31031100 101001,00010,000 n log n306009,000130,000 n2n2 10010,00010 6 10 8 n3n3 1,00010 6 10 9 10 12 2n2n 1,02410 30 10 300 10 3000 n n

20. COSC 3101, PROF. J. ELDER 20 Which are more alike? n 1000 n2n2 2n2n

21. End of Lecture 1 Wed, March 4, 2009

22. COSC 3101, PROF. J. ELDER 22 Which are more alike? Polynomials n 1000 n2n2 2n2n

23. COSC 3101, PROF. J. ELDER 23 Which are more alike? 1000n 2 3n 2 2n 3

24. COSC 3101, PROF. J. ELDER 24 Which are more alike? Quadratic 1000n 2 3n 2 2n 3

25. COSC 3101, PROF. J. ELDER 25 Classifying Functions? Functions

26. COSC 3101, PROF. J. ELDER 26 Classifying Functions Functions Constant Logarithmic Poly Logarithmic PolynomialExponentialExpDouble Exponential (log n) 5 n5n5 2 5n 5 log n5 2 n5n5 2 5n 2

27. COSC 3101, PROF. J. ELDER 27 Classifying Functions? Polynomial

28. COSC 3101, PROF. J. ELDER 28 Classifying Functions Polynomial LinearQuadratic Cubic 4 th Order 5n 2 5n 5n 3 5n 4

29. COSC 3101, PROF. J. ELDER 29 Classifying Functions Functions Constant Logarithmic Poly Logarithmic PolynomialExponentialExpDouble Exponential (log n) 5 n5n5 2 5n 5 log n5 2 n5n5 2 5n 2

30. COSC 3101, PROF. J. ELDER 30 Properties of the Logarithm

31. COSC 3101, PROF. J. ELDER 31 Logarithmic log 10 n = # digits to write n log 2 n = # bits to write n = 3.32 log 10 n log(n 1000 ) = 1000 log(n) Differ only by a multiplicative constant.

32. COSC 3101, PROF. J. ELDER 32 Classifying Functions Functions Constant Logarithmic Poly Logarithmic PolynomialExponentialExpDouble Exponential (log n) 5 n5n5 2 5n 5 log n5 2 n5n5 2 5n 2

33. COSC 3101, PROF. J. ELDER 33 Poly Logarithmic (log n) 5 = log 5 n

34. COSC 3101, PROF. J. ELDER 34 (Poly)Logarithmic << Polynomial For sufficiently large n log 1000 n << n 0.001

35. COSC 3101, PROF. J. ELDER 35 Classifying Functions Polynomial LinearQuadratic Cubic 4 th Order 5n 2 5n 5n 3 5n 4

36. COSC 3101, PROF. J. ELDER 36 Linear << Quadratic For sufficiently large n 10000 n << 0.0001 n 2

37. COSC 3101, PROF. J. ELDER 37 Classifying Functions Functions Constant Logarithmic Poly Logarithmic PolynomialExponentialExpDouble Exponential (log n) 5 n5n5 2 5n 5 log n5 2 n5n5 2 5n 2

38. COSC 3101, PROF. J. ELDER 38 Polynomial << Exponential For sufficiently large n n 1000 << 2 0.001 n

39. COSC 3101, PROF. J. ELDER 39 Ordering Functions Functions Constant Logarithmic Poly Logarithmic PolynomialExponentialExpDouble Exponential (log n) 5 n5n5 2 5n 5 log n5 2 n5n5 2 5n 2 <<

40. COSC 3101, PROF. J. ELDER 40 Which Functions are Constant? 5 1,000,000,000,000 0.0000000000001 -5 0 8 + sin(n) Yes No Yes

41. COSC 3101, PROF. J. ELDER 41 Which Functions are “Constant”? 5 1,000,000,000,000 0.0000000000001 -5 0 8 + sin(n) Yes No Yes Lies in between 7 9 The running time of the algorithm is a “Constant” It does not depend significantly on the size of the input.

42. COSC 3101, PROF. J. ELDER 42 Which Functions are Quadratic? n 2 … ?

43. COSC 3101, PROF. J. ELDER 43 Which Functions are Quadratic? n 2 0.001 n 2 1000 n 2 Some constant times n 2.

44. COSC 3101, PROF. J. ELDER 44 Which Functions are Quadratic? n 2 0.001 n 2 1000 n 2 5n 2 + 3n + 2log n

45. COSC 3101, PROF. J. ELDER 45 Which Functions are Quadratic? n 2 0.001 n 2 1000 n 2 5n 2 + 3n + 2log n Lies in between

46. COSC 3101, PROF. J. ELDER 46 Which Functions are Quadratic? n 2 0.001 n 2 1000 n 2 5n 2 + 3n + 2log n Ignore low-order terms Ignore multiplicative constants. Ignore "small" values of n. Write θ(n 2 ).

47. COSC 3101, PROF. J. ELDER 47 Which Functions are Polynomial? n 5 … ?

48. COSC 3101, PROF. J. ELDER 48 Which Functions are Polynomial? n c n 0.0001 n 10000 n to some constant power.

49. COSC 3101, PROF. J. ELDER 49 Which Functions are Polynomial? n c n 0.0001 n 10000 5n 2 + 8n + 2log n 5n 2 log n 5n 2.5

50. COSC 3101, PROF. J. ELDER 50 Which Functions are Polynomial? n c n 0.0001 n 10000 5n 2 + 8n + 2log n 5n 2 log n 5n 2.5 Lie in between

51. COSC 3101, PROF. J. ELDER 51 Which Functions are Polynomials? n c n 0.0001 n 10000 5n 2 + 8n + 2log n 5n 2 log n 5n 2.5 Ignore low-order terms Ignore power constant. Ignore "small" values of n. Write n θ(1)

52. COSC 3101, PROF. J. ELDER 52 Which Functions are Exponential? 2 n … ?

53. COSC 3101, PROF. J. ELDER 53 Which Functions are Exponential? 2 n 2 0.0001 n 2 10000 n 2 raised to a linear function of n.

54. COSC 3101, PROF. J. ELDER 54 Which Functions are Exponential? 2 n 2 0.0001 n 2 10000 n 8 n 2 n / n 100 2 n · n 100 too small? too big?

55. COSC 3101, PROF. J. ELDER 55 Which Functions are Exponential? 2 n 2 0.0001 n 2 10000 n 8 n 2 n / n 100 2 n · n 100 = 2 3n > 2 0.5n < 2 2n Lie in between

56. COSC 3101, PROF. J. ELDER 56 Which Functions are Exponential? 2 n 2 0.0001 n 2 10000 n 8 n 2 n / n 100 2 n · n 100 = 2 3n > 2 0.5n < 2 2n 2 0.5n > n 100 2 n = 2 0.5n · 2 0.5n > n 100 · 2 0.5n 2 n / n 100 > 2 0.5n

57. COSC 3101, PROF. J. ELDER 57 Which Functions are Exponential? Ignore low-order terms Ignore base. Ignore "small" values of n. Ignore polynomial factors. Write 2 θ(n) 2 n 2 0.0001 n 2 10000 n 8 n 2 n / n 100 2 n · n 100

58. COSC 3101, PROF. J. ELDER 58 Classifying Functions Rank constants in significance. f(n) = 8·2 4n / n 100 + 5·n 3 Tell me the most significant thing about your function

59. COSC 3101, PROF. J. ELDER 59 Classifying Functions f(n) = 8·2 4n / n 100 + 5·n 3 Tell me the most significant thing about your function 2 θ(n) 2 3n << 2 4n f(n)because

60. COSC 3101, PROF. J. ELDER 60 Classifying Functions f(n) = 8·2 4n / n 100 + 5·n 3 Tell me the most significant thing about your function 2 θ(n) 2 4n /n θ(1) θ(2 4n /n 100 ) 8·2 4n / n 100 + θ(n 3 ) 8·2 4n / n 100 + n θ(1) 8·2 4n / n 100 + 5·n 3

61. COSC 3101, PROF. J. ELDER 61 Notations Theta f(n) = θ(g(n))f(n) ≈ c g(n) Big Oh f(n) = O(g(n))f(n) ≤ c g(n) Big Omega f(n) = Ω(g(n))f(n) ≥ c g(n) Little Oh f(n) = o(g(n))f(n) << c g(n) Little Omega f(n) = ω(g(n))f(n) >> c g(n)

62. COSC 3101, PROF. J. ELDER 62 Definition of Theta f(n) = θ(g(n))

63. COSC 3101, PROF. J. ELDER 63 Definition of Theta f(n) is sandwiched between c 1 g(n) and c 2 g(n) f(n) = θ(g(n))

64. COSC 3101, PROF. J. ELDER 64 Definition of Theta f(n) is sandwiched between c 1 g(n) and c 2 g(n) for some sufficiently small c 1 (= 0.0001) for some sufficiently large c 2 (= 1000) f(n) = θ(g(n))

65. COSC 3101, PROF. J. ELDER 65 Definition of Theta For all sufficiently large n f(n) = θ(g(n))

66. COSC 3101, PROF. J. ELDER 66 Definition of Theta For all sufficiently large n For some definition of “sufficiently large” f(n) = θ(g(n))

67. COSC 3101, PROF. J. ELDER 67 Definition of Theta 3n 2 + 7n + 8 = θ(n 2 )

68. COSC 3101, PROF. J. ELDER 68 Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) c 1 ·n 2  3n 2 + 7n + 8  c 2 ·n 2 ??

69. COSC 3101, PROF. J. ELDER 69 Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) 3·n 2  3n 2 + 7n + 8  4·n 2 34

70. COSC 3101, PROF. J. ELDER 70 Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) 3·1 2  3·1 2 + 7·1 + 8  4·1 2 1 False

71. COSC 3101, PROF. J. ELDER 71 Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) 3·7 2  3·7 2 + 7·7 + 8  4·7 2 7 False

72. COSC 3101, PROF. J. ELDER 72 Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) 3·8 2  3·8 2 + 7·8 + 8  4·8 2 8 True

73. COSC 3101, PROF. J. ELDER 73 Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) 3·9 2  3·9 2 + 7·9 + 8  4·9 2 9 True

74. COSC 3101, PROF. J. ELDER 74 Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) 3·10 2  3·10 2 + 7·10 + 8  4·10 2 10 True

75. COSC 3101, PROF. J. ELDER 75 Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) 3·n 2  3n 2 + 7n + 8  4·n 2 ?

76. COSC 3101, PROF. J. ELDER 76 Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) 3·n 2  3n 2 + 7n + 8  4·n 2 n  8 8

77. COSC 3101, PROF. J. ELDER 77 Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) 3·n 2  3n 2 + 7n + 8  4·n 2 True n  8

78. COSC 3101, PROF. J. ELDER 78 Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) 3·n 2  3n 2 + 7n + 8  4·n 2 7n + 8  1 · n 2 7 + 8 / n  1 · n n  8 True

79. COSC 3101, PROF. J. ELDER 79 Definition of Theta 3n 2 + 7n + 8 = θ(n 2 ) 3·n 2  3n 2 + 7n + 8  4·n 2 348 n  8 True

80. COSC 3101, PROF. J. ELDER 80 Asymptotic Notation Theta f(n) = θ(g(n))f(n) ≈ c g(n) BigOh f(n) = O(g(n))f(n) ≤ c g(n) Omega f(n) = Ω(g(n))f(n) ≥ c g(n) Little Oh f(n) = o(g(n))f(n) << c g(n) Little Omega f(n) = ω(g(n))f(n) >> c g(n)

81. COSC 3101, PROF. J. ELDER 81 Definition of Theta 3n 2 + 7n + 8 = θ(n)

82. COSC 3101, PROF. J. ELDER 82 Definition of Theta 3n 2 + 7n + 8 = θ(n) c 1 ·n  3n 2 + 7n + 8  c 2 ·n ??

83. COSC 3101, PROF. J. ELDER 83 Definition of Theta 3n 2 + 7n + 8 = θ(n) 3·n  3n 2 + 7n + 8  100·n 3 100

84. COSC 3101, PROF. J. ELDER 84 Definition of Theta 3n 2 + 7n + 8 = θ(n) 3·n  3n 2 + 7n + 8  100·n ?

85. COSC 3101, PROF. J. ELDER 85 Definition of Theta 3n 2 + 7n + 8 = θ(n) 3·100  3·100 2 + 7·100 + 8  100·100 100 False

86. COSC 3101, PROF. J. ELDER 86 Definition of Theta 3n 2 + 7n + 8 = θ(n) 3·n  3n 2 + 7n + 8  10,000·n 310,000

87. COSC 3101, PROF. J. ELDER 87 Definition of Theta 3n 2 + 7n + 8 = θ(n) 3·10,000  3·10,000 2 + 7·10,000 + 8  10,000·10,000 10,000 False

88. COSC 3101, PROF. J. ELDER 88 Definition of Theta 3n 2 + 7n + 8 = θ(n) What is the reverse statement?

89. COSC 3101, PROF. J. ELDER 89 Understand Quantifiers!!! SamMary BobBeth John Marilyn Monroe FredAnn SamMary BobBeth John Marilyn Monroe FredAnn  b,  Loves(b, MM)  b, Loves(b, MM)  b,  Loves(b, MM)]  b, Loves(b, MM)]

90. COSC 3101, PROF. J. ELDER 90 Definition of Theta 3n 2 + 7n + 8 = θ(n) The reverse statement

91. COSC 3101, PROF. J. ELDER 91 Definition of Theta 3n 2 + 7n + 8 = θ(n) ?

92. COSC 3101, PROF. J. ELDER 92 Order of Quantifiers f(n) = θ(g(n)) ?

93. COSC 3101, PROF. J. ELDER 93 Understand Quantifiers!!! One girl Could be a separate girl for each boy. SamMary BobBeth John Marilin Monro FredAnn SamMary BobBeth John Marilin Monro FredAnn

94. COSC 3101, PROF. J. ELDER 94 Order of Quantifiers No! It cannot be a different c 1 and c 2 for each n. f(n) = θ(g(n))

95. COSC 3101, PROF. J. ELDER 95 Other Notations Theta f(n) = θ(g(n))f(n) ≈ c g(n) BigOh f(n) = O(g(n))f(n) ≤ c g(n) Omega f(n) = Ω(g(n))f(n) ≥ c g(n) Little Oh f(n) = o(g(n))f(n) << c g(n) Little Omega f(n) = ω(g(n))f(n) >> c g(n)

96. COSC 3101, PROF. J. ELDER 96 Definition of “Big Omega”

97. COSC 3101, PROF. J. ELDER 97 Definition of “Big Oh”

98. COSC 3101, PROF. J. ELDER 98 Definition of “Little Omega”

99. COSC 3101, PROF. J. ELDER 99 Definition of “Little Oh”

100. COSC 3101, PROF. J. ELDER 100 Time Complexity Is a Function Specifies how the running time depends on the size of the input. A function mapping “size” of input  “time” T(n) executed.

101. COSC 3101, PROF. J. ELDER 101 Definition of Time?

102. COSC 3101, PROF. J. ELDER 102 Definition of Time # of seconds (machine dependent). # lines of code executed. # of times a specific operation is performed (e.g., addition). These are all reasonable definitions of time, because they are within a constant factor of each other.

103. COSC 3101, PROF. J. ELDER 103 Definition of Input Size A good definition: #bits Examples:

104. COSC 3101, PROF. J. ELDER 104 Time Complexity Is a Function Specifies how the running time depends on the size of the input. A function mapping: –“size” of input  “time” T(n) executed. Or more precisely: –# of bits n needed to represent the input  # of operations T(n) executed.

105. COSC 3101, PROF. J. ELDER 105 Time Complexity of an Algorithm O(n 2 ): For any input size n>n 0, the algorithm takes no more than cn 2 time on every input. Ω(n 2 ): For any input size n>n 0, the algorithm takes at least cn 2 time on at least one input. θ (n 2 ): Do both. The time complexity of an algorithm is the largest time required on any input of size n. (Worst case analysis.)

106. End of Lecture 2 Monday, Mar 9, 2009

107. COSC 3101, PROF. J. ELDER 107 What is the height of tallest person in the class? Bigger than this? Need to find only one person who is taller Need to look at every person Smaller than this?

108. COSC 3101, PROF. J. ELDER 108 Time Complexity of a Problem O(n 2 ): Provide an algorithm that solves the problem in no more than this time. –Remember: for every input, i.e. worst case analysis! Ω(n 2 ): Prove that no algorithm can solve it faster. –Remember: only need one input that takes at least this long! θ (n 2 ): Do both. The time complexity of a problem is the time complexity of the fastest algorithm that solves the problem.