1. Algorithm Course Dr. Aref Rashad February 20131 Algorithms Course..... Dr. Aref Rashad

2. Course Objectives Algorithm Definition and Development Algorithm Complexity Asymptotic Analysis of algorithms Classification of Algorithms Techniques for Algorithm Developments Development and Evaluation of Algorithms for basic processes, as Sorting, Searching …. etc. February 20132 Algorithms Course..... Dr. Aref Rashad

3. February 20133 Algorithms Course..... Dr. Aref Rashad

4. Algorithms and Programs Algorithm: a method or a process followed to solve a problem. – A recipe. An algorithm takes the input to a problem (function) and transforms it to the output. – A mapping of input to output. A problem can have many algorithms, that may differ dramatically in concept, speed and space requirements February 20134 Problem Algorithm Computer OutputInput Algorithms Course..... Dr. Aref Rashad

5. Algorithm Properties An algorithm possesses the following properties: – It must be correct. – It must be composed of a series of concrete steps. – There can be no ambiguity as to which step will be performed next. – It must be composed of a finite number of steps. – It must terminate. A computer program is an instance, or concrete representation, for an algorithm in some programming language. February 20135 Algorithms Course..... Dr. Aref Rashad

6. Example: Polynomial Evaluation Suppose that exponentiation is carried out using multiplications. Two ways to evaluate the polynomial p(x) = 4x 4 + 7x 3 –2x 2 + 3x + 6 are: Brute force method: p(x) = 4*x*x*x*x + 7*x*x*x –2*x*x + 3*x + 6 Horner’s method: p(x) = (((4*x + 7) * x –2) * x + 3) * x + 6 February 20136 Algorithms Course..... Dr. Aref Rashad

7. Algorithm Specification Pseudocode Conventions (English like Statements) -Comments included -Data types are not explicitly declared -Logical operators and, or and not can be used -Relational operators can be used,=,…et -Arrays can be used, e.g. A(i,j) -Looping statements are employed: for, while, and repeat-until -Conditional statements can be used If, then February 20137 Algorithms Course..... Dr. Aref Rashad

8. 1.Initialize an integer, Sum, to zero. 2. For all array values, Increase Sum by array value 3. Go to step 2. February 20138 Algorithms Course..... Dr. Aref Rashad Pesudo Code Algorithm ArraySum(a,n) Sum=0 For i=1 to n do Sum=Sum + a(i) return Sum Problem: Array Sum A Solution: For an array A with length n, sum all the array elements in a new integer An initial Algorithm Translating a Problem into an Algorithm

9. 1. Initialize integer, C, to zero. 2. If A is zero, we’re done and C Contains the result. Otherwise, proceed to step 3. 3. Add the value of B to C. 4. Decrement A. 5. Go to step 2. Pseudocode: Function Multiply(Integer A, Integer B) Integer C = 0 While A is greater than 0 C = C + B A = A - 1 End Return C End February 20139 Algorithms Course..... Dr. Aref Rashad Translating a Problem into an Algorithm Problem: Integers Multiplication A Solution: Given any two integers A and B, we can say that multiplying A times B involves adding B to itself, A times. An initial Algorithm

10. Translating a Problem into an Algorithm February 201310 Problem: Sort a collection of n>=1 elements of arbitrary type A Solution: From those elements that are currently unsorted, find the smallest and place it next in the sorted list An initial Algorithm: For i=1 to n do Examine a(i) to a(n) and suppose the smallest element is at a(j) Interchange a(i) and a(j) Pseudocode: SelectionSort (a,n) For i=1 to n do j=I For k=i+1 to n do if ( a(k) < a(j) ) then j=k t=a(i), a(i)= a(j), a(j)=t Algorithms Course..... Dr. Aref Rashad

11. Analysis of algorithms The theoretical study of computer-program performance and resource usage. What’s more important than performance Modularity Correctness Maintainability Functionality Robustness user-friendliness Programmer time Simplicity Extensibility Reliability February 201311 Algorithms Course..... Dr. Aref Rashad

12. February 2013 Algorithms Course..... Dr. Aref Rashad 12 Functionality: The degree to which the software satisfies stated needs (suitability, accuracy, inter-operability, compliance, and security) Reliability: The amount of time that the software is available for use, (maturity, fault tolerance, and recoverability) Usability: The degree to which the software is easy to use, (understandability, learnability, and operability) Analysis of algorithms

13. February 2013 Algorithms Course..... Dr. Aref Rashad 13 Efficiency: The degree to which the software makes optimal use of system resources (time behavior, resource behavior) Maintainability: The ease with which repair may be made to the software (analyzability, changeability, stability, and testability) Portability: The ease with which the software can be transposed from one environment to another (adaptability, replaceability) Analysis of algorithms

14. February 201314 Why study algorithms and performance? Algorithms help us to understand scalability. Performance often draws the line between what is feasible and what is impossible. Algorithmic mathematics provides a language for talking about program behavior. Performance is the currency of computing. Help to choose between different Algorithms to solve a problem Algorithms Course..... Dr. Aref Rashad

15. Comparing Algorithms: Empirical Approach (1) Implement each candidate  – That could be lots of work – also error-prone (2) Run it  – Which inputs? Test data (3) Time it  – What machines / OS? February 201315 How to solve “which algorithm” problems without machines or test data Algorithms Course..... Dr. Aref Rashad

16. Analytical Approach: February 201316 Problem Algorithm #1 Algorithm #2 … Algorithm #n solveEfficiency Grow proportionally with the amount of data Algorithms Course..... Dr. Aref Rashad

17. February 201317 Algorithm #1 Algorithm #2 … Algorithm #n Efficiency Compare? Computational Complexity : Measure the degree of difficulty of an algorithm Analytical Approach: Algorithms Course..... Dr. Aref Rashad

18. February 201318 Computational Complexity : How much effort ? How costly ? Various ways of measuring Our concern  efficiency criteria of time and space Focus How can we measure time and space? Analytical Approach: Algorithms Course..... Dr. Aref Rashad

19. Space Complexity Measure of an algorithm’s memory requirements during runtime. Data structures, Temporary variables. February 201319 Algorithm ArraySum(a,n) Sum=0 For i=1 to n do Sum=Sum + a(i) return Sum Space needed equal to the sum of: -Fixed part: independent of the Characteristics of Inputs and Outputs (Instruction space, space for variables and constants …etc. -Variable part: dependent on the problem instances Algorithms Course..... Dr. Aref Rashad Focus on estimating Variable part which depends on number and magnitude of Inputs and Outputs

20. The Sum of Compile time and Run time Compile time doesn’t depend on I/O Characteristics February 201320 Time Complexity The running time depends on the input Parameterize the running time by the size of the input Generally, we seek upper bounds on the running time, because everybody likes a guarantee. Algorithms Course..... Dr. Aref Rashad Focus only on Run time

21. Exact Formula: Summation of time needed for all operations (e.g. add, subtract, multiply,….) ………………… Impossible task Experimental Approach: Type, compile and run on a specific machine… Time may differ on multiuser system February 201321 Time Complexity :How to measure? Algorithms Course..... Dr. Aref Rashad

22. Comments ……..… 0 Step Assignment ………... 1 Step Loops ………………….. No of steps account in control part First Method: Introduce a new variable Count into the Program Second Method: Build a table to list total No of steps contributed by each statement February 201322 Time Complexity Algorithms Course..... Dr. Aref Rashad Approximate Approach: Count Program steps (segment independent of Problem Characteristics). Determining No of Steps:

23. February 201323 Algorithm ArraySum(a,n) Count=-0 Sum=0 Count=Count+1 For i=1 to n do Count=Count+1…………………. for For Sum=Sum + a(i), Count=Count+1…..for Assignment Count=Count+1…………………. for last time of For Count=Count+1…………………. For return return Sum Total number of steps 2n+3 Algorithm ArraySum(a,n) Sum=0 For i=1 to n do Sum=Sum + a(i) return Sum Time Complexity First Method: Introduce a new variable Count into the Program Algorithms Course..... Dr. Aref Rashad

24. Second Method: Build a table to list total No of steps contributed by each statement Determine the number of steps per execution (s/e) of the statement and the total number of times (i.e., frequency) each statement is executed. Algorithm ArraySum(a,n) s/e frequency total steps Sum=0 1 1 1 For i=1 to n do 1 n+1 n+1 Sum=Sum + a(i) 1 n n return Sum 1 1 1 Total 2n+3 T(n) = 2n+3 February 201324 Time Complexity Algorithms Course..... Dr. Aref Rashad

25. Example: Polynomial Evaluation Two ways to evaluate the polynomial p(x) = 4x 4 + 7x 3 –2x 2 + 3x+ 6 Brute force method: p(x) = 4*x*x*x*x + 7*x*x*x –2*x*x + 3*x + 6 Horner’s method: p(x) = (((4*x + 7) * x –2) * x + 3) * x + 6 General form of Polynomial p(x) = a 1 + a 2 x 1 + ……..… + a n+1 x n where a n is non-zero for all n >= 0 February 201325 Algorithms Course..... Dr. Aref Rashad

26. February 201326 T(n) = n 2 /2 + n/2 Example: Polynomial Evaluation Pseudocode: Brute force method: 1 Input a 1,…., a n+1 and x 2 Initialize poly = a 1 3 for i = 1,…., n poly = poly + a i+1 * x * x * …..* x (i factors x) end of for loop 4 Output poly Algorithms Course..... Dr. Aref Rashad

27. February 201327 Example: Polynomial Evaluation Pseudocode: Horner’s method: 1 Input a 1,…., a n+1 and x 2 Initialize poly = a n+1 3 for i = 1,…., n poly = poly * x+ a n-i+1 end of for loop 4 Output poly T(n) = 2n Algorithms Course..... Dr. Aref Rashad

28. February 201328 Example: Polynomial Evaluation Algorithms Course..... Dr. Aref Rashad

29. Time Complexity is not dependent solely on the number of inputs and outputs. Three kinds of step account: Best case, worst case, and average case Ex: Sequential search for K in an array of n integers: Begin at first element in array and look at each element in turn until K is found Best case: The first position of the array has K Worst case: The last position in the array has K Average case: n/2 February 201329 Time Complexity: Best case, worst case, and average case Algorithms Course..... Dr. Aref Rashad

30. The Best Case. Normally, we are not interested in the best case, because: It is too optimistic Not a fair characterization of the algorithms’ running time Useful in some rare cases, where the best case has high probability of occurring. The average case. Often we prefer to know the average-case running time. The average case reveals the typical behavior of the algorithm on inputs of size n. Average case estimation is not always possible. February 201330 Algorithms Course..... Dr. Aref Rashad

31. February 201331 The Worst Case. Useful in many real-time applications. Algorithm must perform at least that well. Might not be a representative measure of the behavior of the algorithm on inputs of size n. Note: If we know enough about the distribution of our input we prefer the average-case analysis. If we do not know the distribution, then we must resort to worst-case analysis. Algorithms Course..... Dr. Aref Rashad

32. February 201332 Growth Rate of the Complexity time function Algorithms Course..... Dr. Aref Rashad Upper bound Concept

33. worst-case time? It depends on the speed of our computer: relative speed (on the same machine), absolute speed (on different machines). BIG IDEA: Ignore machine-dependent constants. Look at growth of T(n)as n→∞. February 201333 Time Complexity Algorithms Course..... Dr. Aref Rashad “Asymptotic Analysis” n large

34. February 201334 Algorithm 1 Algorithm 2 Algorithm 3 Algorithm.. T(n) = 2n+3 g(n) = 3n O(n) T(n) = 4n+8 g(n) =5n T(n) = n 2 +8 g(n) =n 2 O(n 2 ) Upper boundNo. of Stepsn large O(n 3 ) O(log n) Algorithms Course..... Dr. Aref Rashad “Asymptotic Analysis”

35. Growth Rates of the Time Complexities of Algorithms with respect to increasing problem sizes. On a machine running at 1 GHz February 201335 Algorithms Course..... Dr. Aref Rashad Time Complexity

36. February 2013 Algorithms Course..... Dr. Aref Rashad 36 Algorithm Matters

37. Big-oh notation indicates an upper bound. How bad things can get – perhaps things are not nearly bad Lowest possible upper bound Asymptotic analysis is a useful tool to help to structure our thinking when n gets large enough Example: linear search: n 2 is an upper bound, but n is the tightest upper bound. February 2013 37 Algorithms Course..... Dr. Aref Rashad The “Big-Oh” Notation Complexity Analysis

38. Calculating Running Time T(n) Use the source/pseudo code Ignore constants Ignore lower order terms Explicitly assume either – The average case (harder to do) – The worst case (easier) Most analysis uses the worst case

39. February 2013 Algorithms Course..... Dr. Aref Rashad 39 Linear-time Loop for x = 1 to n constant-time operation Note: Constant-time mean independent of the input size. 2-Nested Loops  Quadratic for x = 0 to n-1 for y = 0 to n-1 3-Nested Loops  Cubic for x = 0 to n-1 for y = 0 to n-1 for z = 0 to n-1 constant-time operation f(n) = n 3 …….. The number of nested loops determines the exponent constant-time operation

40. February 2013 Algorithms Course..... Dr. Aref Rashad 40 Add independent loops for x = 0 to n-1 constant-time op for y = 0 to n-1 for z = 0 to n-1 constant-time op for w = 0 to n-1 constant-time op f(n) = n + n 2 + n = n 2 + 2n

41. February 2013 Algorithms Course..... Dr. Aref Rashad 41 Non-trivial loops for x = 1 to n for y = 1 to x constant-time operation for z = 1 to n for y = 1 to z for x = 1 to y constant-op

42. EG: 3x 3 + 5x 2 – 9 = O (x 3 ) Doesn’t mean “3x 3 + 5x 2 – 9 equals the function O (x 3 )” Which actually means “3x 3 +5x 2 –9 is dominated by x 3 ” Read as: “3x 3 +5x 2 –9 is big-Oh of x 3 ” February 201342 The “Big-Oh” Notation In fact, 3x 3 +5x 2 –9 is smaller than 5x 3 for large enough values of x Algorithms Course..... Dr. Aref Rashad

43. O-notation (upper bounds): February 201343 So g(n) is an asymptotic upper-bound for f(n) as n increases (g(n) bounds f(n) from above) Algorithms Course..... Dr. Aref Rashad

44. f(n) is O(g(n)) if f grows at most as fast as g. February 201344 cg(n) is an approximation to f(n), bounding from above Algorithms Course..... Dr. Aref Rashad

45. Big-Oh Rules If is f(n) a polynomial of degree d, then f(n) is O(n d ), i.e., Drop lower-order terms Drop constant factors Use the smallest possible class of functions Say “2n is O(n)” instead of “2n is O(n 2 )” Use the simplest expression of the class Say “3n + 5 is O(n)” instead of “3n + 5 is O(3n)”

46. Example 1: If T(n) = 3n 2 then T(n) is in O(n 2 ). Example 2: T(n) = c 1 n 2 + c 2 n in average case. c 1 n 2 + c 2 n 1. T(n) <= cn 2 for c = c 1 + c 2 and n 0 = 1. Therefore, T(n) is in O(n 2 ) by the definition. Example 3: T(n) = c. We say this is in O(1). February 2013 46 Algorithms Course..... Dr. Aref Rashad

47. February 201347 By definition, we need to find: a real constant c > 0 an integer constant n 0 >= 1 Such that 7n - 2 <= c n, for every integer n >= 0 Possible choice is: c = 7 n = 1 7n - 2 = 1 Algorithms Course..... Dr. Aref Rashad

48. February 201348 Algorithms Course..... Dr. Aref Rashad

49. February 201349 Algorithms Course..... Dr. Aref Rashad

50. Set definition of O-notation February 201350 Algorithms Course..... Dr. Aref Rashad

51. Meaning: For all data sets big enough (i.e., n > n 0 ), the algorithm always executes in more than cg(n) steps. February 201351 Algorithms Course..... Dr. Aref Rashad

52. T(n) = c 1 n 2 + c 2 n. c 1 n 2 + c 2 n >= c 1 n 2 for all n > 1. T(n) >= cn 2 for c = c 1 and n 0 = 1. Therefore, T(n) is in  (n 2 ) by the definition. We want the greatest lower bound. February 2013 52 Algorithms Course..... Dr. Aref Rashad

53. February 201353 Algorithms Course..... Dr. Aref Rashad

54. February 201354 When big-Oh and  meet, we indicate this by using  (big-Theta) notation. Definition: An algorithm is said to be  (h(n)) if it is in O(h(n)) and it is in  (h(n)). Algorithms Course..... Dr. Aref Rashad

55. February 201355 Algorithms Course..... Dr. Aref Rashad

56. Relations Between Q, O, W

57. Intuition for Asymptotic Notation Big-Oh f(n) is O(g(n)) if f(n) is asymptotically less than or equal to g(n) big-Omega f(n) is  (g(n)) if f(n) is asymptotically greater than or equal to g(n) big-Theta f(n) is  (g(n)) if f(n) is asymptotically equal to g(n)

58. February 201358 Algorithms Course..... Dr. Aref Rashad

59. February 201359 Algorithms Course..... Dr. Aref Rashad

60. Constant Time: O(1) O(1) actually means that an algorithm takes constant time to run; in other words, performance isn’t affected by the size of the problem. Linear Time: O(N) An algorithm runs in O(N) if the number of operations required to perform a function is directly proportional to the number of items being processed Example: waiting line at a supermarket February 201360 Algorithms Course..... Dr. Aref Rashad The “Big-Oh” Notation

61. February 2013 Algorithms Course..... Dr. Aref Rashad 61 Quadratic Time: O(N 2 ) An algorithm runs in O(N 2 ) if the number of operations required to perform a function is directly proportional to the quadratic number of items being processed Example: Group shakehand

62. Logarithmic Time: O(log N) and O(N log N) The running time of a logarithmic algorithm increases with the log of the problem size. When the size of the input data set increases by a factor of a million, the run time will only increase by some factor of log(1,000,000) = 6. February 201362 Algorithms Course..... Dr. Aref Rashad Factorial Time: O(N!) It is far worse than even O(N 2 ) and O(N 3 ) It’s fairly unusual to encounter functions with this kind of behavior

63. February 201363 Algorithms Course..... Dr. Aref Rashad The “Big-Oh” Notation

64. February 201364 Algorithms Course..... Dr. Aref Rashad

65. Practical Considerations No such big difference in running time between Θ 1 (n) and Θ 2 (nlogn). Θ 1 (10,000) = 10,000 Θ 2 (10,000) =10,000 log 10 10,000 = 40,000 February 2013 65 Algorithms Course..... Dr. Aref Rashad There is an enormous difference between Θ 1 (n 2 ) and Θ 2 (nlogn). Θ 1 (10,000) = 100,000,000 Θ 2 (10,000) = 10,000 log 10 10,000 = 40,000

66. Remarks Most statements in a program do not have much effect on the running time of that program; There is little point to cutting in half the running time of a subroutine that accounts for only 1% of the total. Focus your attention on the parts of the program that have the most impact February 2013 66 Algorithms Course..... Dr. Aref Rashad The greatest time and space improvements come from a better data structure or algorithm “FIRST TUNE THE ALGORITHM, THEN TUNE THE CODE”

67. Remarks When tuning code, it is important to gather good timing statistics; Be careful not to use tricks that make the program unreadable; Make use of compiler optimizations; Check that your optimizations really improve the program. February 2013 67 Algorithms Course..... Dr. Aref Rashad

68. An algorithm with time equation T(n) = 2n 2 does not receive nearly as great an improvement from the faster machine as an algorithm with linear growth rate. Instead of an improvement by a factor of ten, the improvement is only the square root of 10 (≈3.16) Instead of buying a faster computer, consider what happens if you replace an algorithm with quadratic running time with a new algorithm with n logn running time February 201368 Remarks Algorithms Course..... Dr. Aref Rashad

69. February 2013 Algorithms Course..... Dr. Aref Rashad 69 Lecture 1 End

70. Comparison of Functions f  g  a  b f (n) = O(g(n))  a  b f (n) =  (g(n))  a  b f (n) =  (g(n))  a = b f (n) = o(g(n))  a < b f (n) =  (g(n))  a > b

71. Limits lim [f (n) / g (n)] = 0  f (n)   (g (n)) n  lim [f (n) / g (n)] <   f (n)   (g (n)) n  0 < lim [f (n) / g (n)] <   f (n)   (g (n)) n  0 < lim [f (n) / g (n)]  f (n)  (g (n)) n  lim [f (n) / g (n)] =   f (n)   (g (n)) n  lim [f (n) / g (n)] undefined  can’t say n 

72. Logarithms x = log b a is the exponent for a = b x. Natural log: ln a = log e a Binary log: lg a = log 2 a lg 2 a = (lg a) 2 lg lg a = lg (lg a)

73. Logarithms and exponentials If the base of a logarithm is changed from one constant to another, the value is altered by a constant factor. Ex: log 10 n * log 2 10 = log 2 n. Base of logarithm is not an issue in asymptotic notation. Exponentials with different bases differ by a exponential factor (not a constant factor). Ex: 2 n = (2/3) n *3 n.

74. Examples Express functions in A in asymptotic notation using functions in B. A B 5n 2 + 100n 3n 2 + 2 A   (n 2 ), n 2   (B)  A   (B) log 3 (n 2 ) log 2 (n 3 ) log b a = log c a / log c b; A = 2lgn / lg3, B = 3lgn, A/B =2/(3lg3) n lg4 3 lg n a log b = b log a ; B =3 lg n =n lg 3 ; A/B =n lg(4/3)   as n   lg 2 n n 1/2 lim ( lg a n / n b ) = 0 (here a = 2 and b = 1/2)  A  o (B) n  A   (B) A   (B) A  o (B)

75. Math Basics Constant Series: For integers a and b, a  b, Linear Series (Arithmetic Series): For n  0, Quadratic Series: For n  0,

76. Cubic Series: For n  0, Geometric Series: For real x  1, For |x| < 1, Math Basics

77. Linear-Geometric Series: For n  0, real c  1, Harmonic Series: nth harmonic number, n  I +, Math Basics

78. Telescoping Series: Differentiating Series: For |x| < 1, Math Basics

79. Many problems whose obvious solution requires Θ(n 2 ) time also has a solution that requires Θ(nlogn). Examples: – Sorting – Searching February 2013 79 Algorithms Course..... Dr. Aref Rashad Practical Considerations

80. Remarks Comparative timing of programs is a difficult business: – Experimental errors from uncontrolled factors (system load, language, compiler etc..); – Bias towards a program; – Unequal code tuning. February 2013 80 Algorithms Course..... Dr. Aref Rashad

81. February 201381 Algorithms Course..... Dr. Aref Rashad

82. February 201382 Algorithms Course..... Dr. Aref Rashad

83. February 201383 Algorithms Course..... Dr. Aref Rashad

84. February 201384 Algorithms Course..... Dr. Aref Rashad

85. February 201385 Algorithms Course..... Dr. Aref Rashad

86. February 201386 Algorithms Course..... Dr. Aref Rashad

87. February 201387 Algorithms Course..... Dr. Aref Rashad

88. February 201388 Algorithms Course..... Dr. Aref Rashad

89. Faster Computer or Algorithm? What happens when we buy a computer 10 times faster T(n)T(n)nn’n’Changen’/n 10n1,00010,000n’ = 10n10 20n5005,000n’ = 10n10 5n log n2501,842  10 n < n’ < 10n 7.37 2n22n2 70223 n’ =  10n 3.16 2n2n 1316n’ = n + 3----- February 201389 Algorithms Course..... Dr. Aref Rashad

90. February 201390 Implications of Dominance Exponential algorithms get hopeless fast. Quadratic algorithms get hopeless at or before 1,000,000. O(n log n) is possible to about one billion. O(log n) never sweats. Implications of Dominance na dominates nb if a > b since lim n!1 nb=na = nba ! 0 na + o(na) doesn’t dominate na since lim n!1 na=(na + o(na)) ! 1 Algorithms Course..... Dr. Aref Rashad

91. February 201391 Algorithms Course..... Dr. Aref Rashad